Executive Summary

What have we done?

In this impact evaluation, we study the effect of the Tá de Pé App and Campaign. Tá de Pé (TDP) is an initiative to improve the responsiveness in Education Infrastructure public expenditures. From August 2017 to February 2019, we studied the impact of the App and the impact of an online email campaign designed to pressure all municipalities known to have delayed school constructions according to the Brazilian Federal government Ministry of Education.

Why should we care?

Despite being among the ten largest economies in the world, Brazil scores consistently in the last positions at the PISA education exam. Many public school constructions in Brazil are delayed or unfinished due to inefficiencies and corruption. This impacts the long-run productivity and welfare in the country and limits the returns of investments in the country.

How did we do?

We evaluated three interventions:

  1. The effect of the App introduction, from August 2017 to February 2018.
  2. The effect of the App introduction, from August 2018 to February 2019.
  3. The effect of the TDP Campaign, where the schools with delayed constructions received an automated email requesting information, in December 2018.

To access the impact, we randomize a set of schools and municipalities not to receive the intervention. We then studied construction execution statistics, construction status, and construction finishing dates.

What did we find?

The first intervention found that the introduction of the App increased the likelihood of school completion by 6.85 percent. The second intervention found that the App pressure over the bureaucracy increased the investment in the schools by 157.49 percent. The last intervention had a null impact on school constructions statistics.

How can we improve?

For further development, we suggest the following:

  1. Gamification of the App, to attract younger users: younger people have more time to use the App. Focusing on this group may improve App usage and the end-line impact on school constructions. For example, a partnership with Pokemon Go, to put Pokemons over the school constructions, together with banners embedded in the game that incentivize reporting, could increase the end-line impact.

  2. Expansion of the intervention scope: school construction is only part of a much broader set of educational services provided. School quality and teacher attendance are also essential and problematic in Brazil. Moreover, the design can be easily transported to other fields, such as health care. This could make the App a comprehensive whistleblowing tool for both education and health care systems.

  3. Shocks in the cost-benefits of using the App: improve the cost-benefit equation could increase the potential number of users. This can be done by providing Tinder, Rappi, Uber, or Google Play credits for users that effectively report.

The TDP Project

The Tá de Pé project is an initiative carried out by the Transparência Brasil (Brazilian Transparency) to foster bottom-up accountability. The App was built to facilitate people to go to school construction sites, check their status, and request information to the public authorities about the steps toward finishing the school constructions.

The app works as follows. From a list of constructions available in the Ministry of Education website, the Brazilian Transparency mapped all the schools and made them available in the Ta de Pé App. Any person with the App can go to a construction site and take pictures. The Ta de Pé bot sends the images for engineers to evaluate the execution and suggest whether to report the construction site as delayed. If the construction is reported, then the App sends a request to the mayor’s office asking the public officials to explain why the building is unfinished and requesting better completion estimates.

The App has received the 2016 Google Social Impact grant, with more than 200 thousand popular votes and it has been online since April 2017.

Impact evaluation design

We divided the impact evaluation into three interventions. The first two interventions will evaluate the impact of the App on reported school constructions outcomes. The last intervention will assess the result of a campaign run by the Tá de Pé bot, that reported all the school constructions marked as delayed in the Ministry of Education dataset.

Randomization

In the first intervention, we use simple randomization, placing 150 municipalities in the control group. In the second and third interventions, we use block randomization, randomizing by three characteristics: Brazilian State, Construction Status (delayed, ongoing, stopped), and above median executed indicator.

To evaluate the random assignment, we used the following pre-treatment variables:

  1. Log of Municipal Population in 2015
  2. Log of Number of Poor Families (2010 IBGE Census)
  3. Log of Total Federal Transfers to the Municipality in 2016
  4. IDEB Indicator for primary school quality (2015 Ministry of Education)
  5. IDEB Indicator for secondary school quality (2015 Ministry of Education)

For the first randomization, we used the core R function sample. For the block randomization, we use the package randomizr for R. We take the approach of a small control group, as this was a sensitive condition to not prevent the app from being evaluated. In the first intervention, we select 150 municipalities. In the second and third interventions, we selected around 600 construction sites. The difference in the randomization procedure is due to better information about construction sites during the second and third interventions.

Qualitatively, the intervention in each step consists of the following:

  1. Intervention 1: We selected 344 construction sites to not appear in the app.

  2. Intervention 2: We selected 656 construction sites to not appear in the app.

  3. Intervention 3: We select 523 construction sites to not receive the TDP bot campaign.

Outcomes

We investigated the impact on six outcomes of interest:

  1. Percentage of the investment executed before the impact evaluation started (placebo)
  2. Percentage of the investment performed by the end of the impact evaluation period
  3. The difference of the percentage invested in the end and at the beginning of the impact evaluation period
  4. Indicator for a construction finished during the impact evaluation period
  5. Indicator for a construction canceled during the impact evaluation period
  6. Indicator for an updated conclusion date for the construction during the impact evaluation period

Our hypothesis, conditional on the app having a positive effect in terms of welfare, is to find that:

  1. The percentage of the investment executed before the impact evaluation started should remain unchanged
  2. The percentage of the investment executed by the end of the impact evaluation period should increase, meaning that the App is speeding up the construction.
  3. The difference of the percentage invested in the end and at the beginning of the impact evaluation period should increase meaning that the app is increasing the amount invested in the project.
  4. The indicator for a construction finished during the impact evaluation period should increase meaning that constructions are being finished at a higher rate.
  5. The indicator for a construction canceled during the impact evaluation period should increase is ambiguous, as it is good to cancel constructions that were not started, but not as much to cancel constructions that are half-way through the work.
  6. The indicator for an updated conclusion date for the construction during the impact evaluation period should increase, meaning that the public officials are giving better estimates of the finishing times.

Manipulation

To check the manipulation, we look at the number and places of app download.

Estimation

For the estimation, we use the following regression equation:

\[ Y_i \ = \ \alpha + \beta T_i + \gamma X_i + \theta Z_i + \varepsilon_i \]

\(i\) indexes the case. \(Y_i\) in an outcome, as described in the previous section. \(\beta\) is the quantity of interest (Average Treatment Effect). \(T_i\) is the treatment that in our case, is an intervention occurrence indicator with two values, zero (no intervention) and one (intervention). \(\gamma\) is a vector of fixed effects, and \(X_i\) is a matrix of Brazilian states’ fixed effects. \(\theta\) is a vector of controls and \(Z_i\) an array of controls for the case \(i\). \(\varepsilon_i\) is the error term.

We cluster the standard errors at the municipal level, as the investment decisions are taken by the mayor’s offices.

As robustness, we fit two extra models. First, we re-run the main model where we have the following: (i) we ran a model without controls or fixed effects; (ii) we run the models adding the control variables, which are the same as the ones used in the covariate balance tests; (iii) we run the regression model with State fixed effects. The results are in the Appendix.

Second, we run the main models using inverse probability weights where we used block randomization and a one-to-one nearest neighborhood matching. We use the control variables as matching characteristics. These robustness checks are intended to correct the imbalance caused by the minimal control group approach that we employed in all three interventions.

Partial implementation and potential issues in the causal identification

In the first intervention, the municipalities in the control group were displayed in the treatment group for about two weeks, in January 2018. When the mistake was detected, the towns were removed promptly. A few municipalities have App activity, but controlling and/or excluding these municipalities from the estimation does not change the results. Note also that this problem lowers the chance to detect an existant effect, making the estimator more conservative.

In the second and third interventions, there were no threats to the identification, with the treatment being fully implemented.

Intervention 1

The intervention one was designed to evaluate the impact of launching the first version of the App. The first version was built for Android only, and was launched in August 2017 and lasted until February 2018.

Randomization

The randomization was performed at the municipal level. We selected 150 municipalities for the control group while the rest were placed in the treatment group. The control group worked by deleting all the construction sites in control assigned cities.

In the next Table, we display the balance test, showing that the covariates are balanced in the treatment and control groups.

## 
## --------Summary descriptives table by 'treatOriginal'---------
## 
## __________________________________________________________________ 
##                                  0              1        p.overall 
##                                N=150          N=1030               
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ 
## Log Population (2015)      10.189 (1.079) 10.172 (1.096)   0.858   
## Log Poor Families (2010)   7.772 (1.022)  7.780 (1.040)    0.926   
## Log Total Transfers (2016) 15.854 (0.832) 15.845 (0.844)   0.903   
## IDEB Inicial Years (2015)  4.973 (0.963)  4.970 (0.979)    0.978   
## IDEB Final Years (2015)    4.015 (0.837)  3.992 (0.804)    0.757   
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

As expected, in the treatment and the control, there are no statistically significant differences in terms of Population in 2015, the number of Poor Families in the 2010 IBGE Census, total and FUNDEB transfers to the municipality in 2016, and the IDEB education quality indicators.

Manipulation

Manipulation in experiments is defined as the extent through which the treatment was successfully delivered. In the case of the App intervention, it represents whether having the App improved the results in terms of school construction completion rates. We present two manipulation indicators. First, we look into the municipalities where we had a successful download and use of the App.

Second, we look into App downloads during the intervention 1 period in the following graph.

The map displays a good territorial variability. There were 455 downloads in the 1020 municipalities originally in the dataset at the beginning of the impact evaluation. Downloads peak during the Facebook TDP campaign, right after the App launch in August, then diminishes over time.

In terms of manipulation, there were downloads in 44.6% of the municipalities, and the downloads were temporally concentrated. This indicates that the Facebook campaign worked, but in terms of manipulation, represents a weak treatment delivery. This because the step-by-step usage of the App involves:

  1. Download the App.
  2. Find a school nearby.
  3. Go to school and take pictures.

As the App download is the first step, this represents low manipulation.

Results

The intervention was sufficient to improve the school completion rate. It increased in 6.85 percent the chance that a school construction becomes reported as finished by the end of the intervention. All other coefficients were not statistically significant at conventional levels.

This finding is positive in the sense that bottom-up pressure exerted by the App worked to increase the completion rate of constructions.

Results using nearest neighborhood matching

In the randomization, we adopted a minimal control group approach. This has implications for statistical power and sensitivity of the results, as much of the variation is placed on the treatment versus the control group. To increase the reliability of the results, we run the main using one-to-one Nearest Neighborhood Matching with Mahalanobis distance. The findings follow below.

As we can see, the impact of the App on the school construction completion resists the robustness checks. This reinforces the reliability of this effect.

Discussion

The introduction of App had a positive effect on the school construction completion rates.

Intervention 2

In intervention 2, we study the impact of the presence of the App on school construction outcomes. Intervention 2 is similar to intervention 1 in all but three characteristics. First, the TDP App was now available for iOS devices. Second, the randomization is now performed at the school level, blocking by characteristics noted below. Finally, the intervention period is August 2018 to February 2019.

Randomization

We randomize the App at the school level, placing 15% of schools that could potentially appear in the App in the control group. We used block randomization, blocking by four characteristics:

  1. The Brazilian State
  2. Status (Under construction, stopped, unfinished)
  3. Above median spent resources

The blocks and the inverse probability weights are located in the Appendix. The next table shows that the randomization is balanced by the municipal pre-treatment variables.

## 
## --------Summary descriptives table by 'treatApp'---------
## 
## __________________________________________________________________ 
##                               Control       Treatment    p.overall 
##                                N=654          N=3895               
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ 
## Log Population (2015)      10.497 (1.540) 10.501 (1.449)   0.956   
## Log Poor Families (2010)   8.092 (1.431)  8.087 (1.319)    0.930   
## Log Total Transfers (2016) 16.095 (1.226) 16.087 (1.136)   0.875   
## IDEB Inicial Years (2015)  4.886 (0.998)  4.908 (0.977)    0.606   
## IDEB Final Years (2015)    3.889 (0.734)  3.891 (0.713)    0.949   
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

Manipulation

In terms of manipulation, we analyze two outcomes: municipality App downloads and downloads over time.

Downloads by municipality:

## Joining, by = "City"

Downloads over time – Intervention 2:

The total number of municipalities that had downloads of the App is 433. There are two spikes in the usage of the App. One in August, right after the intervention start and a second spike around December. The first spike is a result of a TDP social media campaign for boosting App usage. The second spike is a spillover from the TDP reporting campaign, here known as the intervention 3.

The number of downloads is considerably smaller in this second intervention when compared with intervention 1. As we discussed, this represents a weak manipulation, making detecting treatment effects a hard task.

Results from the full model

The full model shows us null effects in all the studied outcomes.

Results using inverse probability weights

As we used block randomization, the treatment effect is only corrected estimated when we use the inverse probability weights to reweight the outcomes. The results follow in the table below, showing null effects.

Results using nearest neighborhood matching

Using nearest neighborhood matching the results improve a little, and we can see that the App improved the difference between the baseline and the end-line reported investment by 157.49 percent.

Although there were no completion effects as in intervention 1, we had a substantial impact investment in the nearest neighborhood matched sample.

Discussion

We have null treatment effects in all but one outcome, in the nearest neighborhood matched sample. In the NN-matching, we find that the App improves the percentage of the executed budget.

Intervention 3

Intervention 3 focused on the effect of a TDP Campaign to boost response, in a top-down email campaign to select schools in December 2018.

Randomization

We randomize the TDP Campaign, placing 15% of schools that could potentially receive the email campaign advocacy in the control group. We used block randomization, blocking by four characteristics:

  1. The Brazilian State
  2. Status (Under construction, stopped, unfinished)
  3. Above median spent resources

The blocks and the inverse probability weights are located in the Appendix. The next table shows that the randomization is balanced by the municipal pre-treatment variables.

## 
## --------Summary descriptives table by 'treatCampaign'---------
## 
## __________________________________________________________________ 
##                               Control       Treatment    p.overall 
##                                N=521          N=3151               
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ 
## Log Population (2015)      10.630 (1.488) 10.525 (1.509)   0.137   
## Log Poor Families (2010)   8.218 (1.365)  8.145 (1.344)    0.256   
## Log Total Transfers (2016) 16.181 (1.163) 16.123 (1.187)   0.288   
## IDEB Inicial Years (2015)  4.812 (0.924)  4.783 (0.929)    0.523   
## IDEB Final Years (2015)    3.824 (0.684)  3.845 (0.699)    0.581   
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

Manipulation

In the manipulation, let us analyze the ex-post effect by looking into responses.

First, we have the following chart for the validity of the responses. We coded the validity of responses in three categories. First, a valid answer provides a minimal explanation that covers the inquiries made by the campaign email. An invalid answer represents an answer without any explanation. Not answer means that the campaign email has not been addressed by the :

As we can see, the state with the most valid responses was the Rio Grande do Sul, while the state with the most significant number of non-response was Maranhao. Maranhao is also the state with most school constructions in the Campaign, which can partially explain their rates. Para is a positive outlier in this analysis, as it has the second-largest number of schools in the campaign, but it managed to answer almost half of the requests with valid answers. Amapa is a negative outlier in the analysis, as no campaign emails have been answered.

The campaign has a higher manipulation impact, which is expected since it consists of sending emails and monitoring responses. One caveat is that the advantage of the App is that it has a more personal component attached to it. Combining these elements could be a good alternative here.

Results from the full model

In the model below, we show that the campaign, despite the sizeable manipulation, had null effects on the studied outcomes.

Results using inverse probability weights

Using inverse probability weights given by the block randomization, we show again that the campaign had null effects on the studied outcomes.

Results using nearest neighborhood matching

Finally, using the nearest neighborhood matching, we still have a consistent null effect of the campaign on the studied outcomes.

Discussion

The campaign had a consistent null effect across all treatments. This goes per most email campaigns aimed at improving development outcomes.

Conclusion

In this report, we evaluated the TDP App (interventions 1 and 2) and the TDP campaign (intervention 3). We find that the campaign has a consistent null effect across all the analysis.

In terms of suggestions, for the first two interventions, we strongly suggest measures to improve the manipulation, such as gamification of the App and shocks in the cost-benefit calculations (such as bonuses for using the App, discount coupons in widely used Apps, and others). For the third intervention, a recent paper by Larreguy et al. (2019) showed that repeating Facebook boosting campaign has a strong effect, as saturation is key for achieving positive results in terms of online campaigns. Therefore, repeating the intervention for small periods of time (i.e., monthly, could improve the effects).

Appendix

In this appendix, we put the APSA Experimental Section report.

APSA Experimental Section Standard Report for Experimental Research

A. Hypothesis

  • Main intervention questions: The experiment studies two problems. First, whether the introduction of the Ta de Pe App improved the governmental school constructions in Brazil. Second, whether a top-down campaign to pressure the bureaucracies to deliver the school infrastructure enhanced outcomes. Therefore, the project here studies how technology facilitates bottom-up and top-down pressure to improve service provision.

  • Hypothesis: There is a set of findings showing that bottom-up peer pressure improves outcomes in developing economies. The main work in the field shows positive peer-pressure effects:

    • Bjorkman, Martina and Jakob Svensson. 2009. “Power to the People: Evidence from a randomized field experiment of a community-based monitoring project in Uganda.” Quarterly Journal of Economics 124(2):735–769.

    • Bjorkman, Martina and Jakob Svensson. 2010. “When is community-based monitoring effective? Evidence from a randomized experiment in primary health in Uganda.” Journal of the European Economic Association 8(2-3):571–581.

    • Bjorkman Nyqvist, Martina, Damien de Walque and Jakob Svensson. 2017. “Experimental evidence on the long-run impact of community-based monitoring.” American Economic Journal: Applied Economics 9(1):33–69.

However, a set of recent papers set out to recheck these hypotheses, finding an overall null result:

This project is essential to reevaluate this dispute and try to propose new means to understand how bottom-up accountability works. First, we are conducting this experiment in a middle-income country, that within the country has municipalities with social indicators close to developed European countries in some areas and developing economies such as African countries, in some other areas. Second, we are evaluating school constructions, and the presence of schools have essential consequences in terms of long-run economic development, the household income composition, and woman empowerment.

B. Subjects and Context

  • Eligibility and Exclusion criteria: We selected all school projects that received Federal fundings from the Brazilian Ministry of Education. By an agreement with the Brazilian Ministry of Education, they allowed us to have the data of all school constructions receiving funds from them.

All schools constructions funded by the Brazilian Ministry of Education participated in this study. We had no schools excluded from the program.

  • Interventions’ dates: We had three interventions here:
  1. App impact evaluation 1: From August 2017 to February 2018. We call this Intervention 1 throughout the text.
  2. App impact evaluation 2: From August 2018 to February 2019. We call this Intervention 2 throughout the text.
  3. Top-down pressure by email impact evaluation: In December 2018, the TDP bot sent an email to all delayed school constructions in the dataset.

C. Allocation Methods

  • Assignment procedure: The treatment here was assigned by the municipality in the first intervention, and at the school level in the second and third interventions.
  1. For intervention one, we selected 150 municipalities that were placed in the control group.

The randomization code was:

vectreat <- c(rep(0,150), rep(1, nrow(dat)-150))

AIC_info = numeric()
for (i in 1:5000) {
  vectreat <- sample(vectreat, length(vectreat))
  mod <- glm(formula(paste('vectreat', vars, sep = '~')), 
             family = binomial, data = dat)
  if(sum(summary(mod)$coefficients[-1,4]<.2)==0) {
    dat = data.frame(dat, vtreat = vectreat)
    AIC_info = c(AIC_info, AIC(mod))
  }
}

And then we checked whether the randomization had one of the covariates selected here significantly predicting the random assignment. We presented a vector of acceptable random assignment in these grounds, and the Brazilian Transparency selected the one we are using in the first intervention.

  1. For interventions 2 and 3, we selected 15% of the schools in the dataset that could either be placed in the treatment or in control in each of the interventions.

In interventions 1 and 2, the schools in the control group did not show up in the App. In the intervention 3, the mayors’ office has not received emails from the Ta de Pe bot.

The randomizr code for intervention 2 follows below. OnApp means treatment and OffApp means the control for this intervention:

decl_intheapp <- declare_ra(blocks = dat_intheapp$var_blocking, 
                            prob_each = c(.85,.15),
                            conditions = c('OnApp', 'OffApp'))

dat_intheapp$Z_intheapp <- block_ra(blocks = dat_intheapp$var_blocking, 
                       conditions = c('OnApp', 'OffApp'),
                       prob_each = c(.85,.15))

dat_intheapp$IPW_intheapp <- 1/obtain_condition_probabilities(decl_intheapp, 
                                                              dat_intheapp$Z_intheapp)

The randomizr code for intervention 3 follows below. Campaign means treatment and ControlCampaign means the control for this intervention:

decl_campaign <- declare_ra(blocks = dat_campaign$var_blocking, 
                          prob_each = c(.85,.15),
                          conditions = c('Campaign', 'ControlCampaign'))

dat_campaign$Z_campaign <- block_ra(blocks = dat_campaign$var_blocking, 
              conditions = c('Campaign', 'ControlCampaign'),
              prob_each = c(.85,.15))

dat_campaign$IPW_campaign <- 1/obtain_condition_probabilities(decl_campaign, 
                                                              dat_campaign$Z_campaign)

*Block randomization: We used simple randomization in the first assignment. In the second and third interventions, we blocked by the Brazilian State, the construction status, and above the median execution. The summary of the blocks, with the IPWs assigned follow below.

Block randomization – Intervention 2
state status above_median_executed var_blocking Z_campaign IPW_campaign
1 CE Unfinished AboveMedianExecuted CE_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
2 PI Unfinished AboveMedianExecuted PI_Unfinished_AboveMedianExecuted Campaign 1.176471
3 PI Unfinished AboveMedianExecuted PI_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
4 MA Unfinished AboveMedianExecuted MA_Unfinished_AboveMedianExecuted Campaign 1.176471
5 RN Unfinished AboveMedianExecuted RN_Unfinished_AboveMedianExecuted Campaign 1.176471
6 TO Unfinished AboveMedianExecuted TO_Unfinished_AboveMedianExecuted Campaign 1.176471
7 TO Unfinished Below_Median_Executed TO_Unfinished_Below_Median_Executed Campaign 1.176471
8 SP Ongoing AboveMedianExecuted SP_Ongoing_AboveMedianExecuted Campaign 1.176471
9 PB Unfinished Below_Median_Executed PB_Unfinished_Below_Median_Executed ControlCampaign 6.666667
10 PB Ongoing AboveMedianExecuted PB_Ongoing_AboveMedianExecuted Campaign 1.176471
13 MG Unfinished AboveMedianExecuted MG_Unfinished_AboveMedianExecuted Campaign 1.176471
14 GO Unfinished AboveMedianExecuted GO_Unfinished_AboveMedianExecuted Campaign 1.176471
15 RJ Unfinished AboveMedianExecuted RJ_Unfinished_AboveMedianExecuted Campaign 1.176471
17 ES Ongoing AboveMedianExecuted ES_Ongoing_AboveMedianExecuted Campaign 1.176471
18 PI Ongoing AboveMedianExecuted PI_Ongoing_AboveMedianExecuted Campaign 1.176471
19 CE Unfinished AboveMedianExecuted CE_Unfinished_AboveMedianExecuted Campaign 1.176471
21 CE Unfinished Below_Median_Executed CE_Unfinished_Below_Median_Executed Campaign 1.176471
24 MG Unfinished AboveMedianExecuted MG_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
26 BA Unfinished AboveMedianExecuted BA_Unfinished_AboveMedianExecuted Campaign 1.176471
30 RJ Unfinished Below_Median_Executed RJ_Unfinished_Below_Median_Executed Campaign 1.176471
33 GO Ongoing AboveMedianExecuted GO_Ongoing_AboveMedianExecuted Campaign 1.176471
36 PB Unfinished Below_Median_Executed PB_Unfinished_Below_Median_Executed Campaign 1.176471
37 RS Unfinished Below_Median_Executed RS_Unfinished_Below_Median_Executed Campaign 1.176471
39 RS Unfinished AboveMedianExecuted RS_Unfinished_AboveMedianExecuted Campaign 1.176471
41 SP Unfinished AboveMedianExecuted SP_Unfinished_AboveMedianExecuted Campaign 1.176471
48 MS Unfinished AboveMedianExecuted MS_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
50 MT Unfinished AboveMedianExecuted MT_Unfinished_AboveMedianExecuted Campaign 1.176471
51 ES Ongoing Below_Median_Executed ES_Ongoing_Below_Median_Executed Campaign 1.176471
52 MG Unfinished Below_Median_Executed MG_Unfinished_Below_Median_Executed ControlCampaign 6.666667
53 MG Stopped AboveMedianExecuted MG_Stopped_AboveMedianExecuted Campaign 1.176471
55 RS Ongoing AboveMedianExecuted RS_Ongoing_AboveMedianExecuted Campaign 1.176471
58 TO Unfinished AboveMedianExecuted TO_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
59 PR Unfinished Below_Median_Executed PR_Unfinished_Below_Median_Executed Campaign 1.176471
61 PB Unfinished AboveMedianExecuted PB_Unfinished_AboveMedianExecuted Campaign 1.176471
64 RN Unfinished AboveMedianExecuted RN_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
68 GO Unfinished AboveMedianExecuted GO_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
73 PI Unfinished Below_Median_Executed PI_Unfinished_Below_Median_Executed Campaign 1.176471
74 MG Unfinished Below_Median_Executed MG_Unfinished_Below_Median_Executed Campaign 1.176471
84 MT Unfinished AboveMedianExecuted MT_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
85 BA Ongoing AboveMedianExecuted BA_Ongoing_AboveMedianExecuted Campaign 1.176471
88 BA Ongoing AboveMedianExecuted BA_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
89 BA Stopped AboveMedianExecuted BA_Stopped_AboveMedianExecuted ControlCampaign 6.666667
90 PR Stopped AboveMedianExecuted PR_Stopped_AboveMedianExecuted ControlCampaign 6.666667
94 PA Ongoing AboveMedianExecuted PA_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
95 PA Stopped AboveMedianExecuted PA_Stopped_AboveMedianExecuted Campaign 1.176471
97 PR Stopped AboveMedianExecuted PR_Stopped_AboveMedianExecuted Campaign 1.176471
100 PA Ongoing Below_Median_Executed PA_Ongoing_Below_Median_Executed Campaign 1.176471
105 RR Ongoing AboveMedianExecuted RR_Ongoing_AboveMedianExecuted Campaign 1.176471
106 AM Unfinished AboveMedianExecuted AM_Unfinished_AboveMedianExecuted Campaign 1.176471
108 AM Unfinished Below_Median_Executed AM_Unfinished_Below_Median_Executed Campaign 1.176471
112 AM Unfinished Below_Median_Executed AM_Unfinished_Below_Median_Executed ControlCampaign 6.666667
116 PA Ongoing AboveMedianExecuted PA_Ongoing_AboveMedianExecuted Campaign 1.176471
118 PA Ongoing Below_Median_Executed PA_Ongoing_Below_Median_Executed ControlCampaign 6.666667
119 PA Stopped AboveMedianExecuted PA_Stopped_AboveMedianExecuted ControlCampaign 6.666667
122 PA Stopped Below_Median_Executed PA_Stopped_Below_Median_Executed Campaign 1.176471
129 CE Ongoing AboveMedianExecuted CE_Ongoing_AboveMedianExecuted Campaign 1.176471
132 TO Ongoing AboveMedianExecuted TO_Ongoing_AboveMedianExecuted Campaign 1.176471
136 TO Stopped AboveMedianExecuted TO_Stopped_AboveMedianExecuted Campaign 1.176471
137 RR Stopped AboveMedianExecuted RR_Stopped_AboveMedianExecuted Campaign 1.176471
139 RR Stopped Below_Median_Executed RR_Stopped_Below_Median_Executed ControlCampaign 6.666667
140 RR Stopped Below_Median_Executed RR_Stopped_Below_Median_Executed Campaign 1.176471
141 RR Ongoing Below_Median_Executed RR_Ongoing_Below_Median_Executed Campaign 1.176471
145 AP Unfinished AboveMedianExecuted AP_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
146 AP Unfinished AboveMedianExecuted AP_Unfinished_AboveMedianExecuted Campaign 1.176471
149 AP Unfinished Below_Median_Executed AP_Unfinished_Below_Median_Executed Campaign 1.176471
153 AP Unfinished Below_Median_Executed AP_Unfinished_Below_Median_Executed ControlCampaign 6.666667
157 GO Ongoing Below_Median_Executed GO_Ongoing_Below_Median_Executed Campaign 1.176471
158 GO Stopped AboveMedianExecuted GO_Stopped_AboveMedianExecuted ControlCampaign 6.666667
163 GO Ongoing Below_Median_Executed GO_Ongoing_Below_Median_Executed ControlCampaign 6.666667
166 GO Stopped AboveMedianExecuted GO_Stopped_AboveMedianExecuted Campaign 1.176471
169 GO Stopped Below_Median_Executed GO_Stopped_Below_Median_Executed Campaign 1.176471
174 MA Ongoing AboveMedianExecuted MA_Ongoing_AboveMedianExecuted Campaign 1.176471
175 MA Ongoing Below_Median_Executed MA_Ongoing_Below_Median_Executed ControlCampaign 6.666667
176 MA Ongoing Below_Median_Executed MA_Ongoing_Below_Median_Executed Campaign 1.176471
186 MA Unfinished Below_Median_Executed MA_Unfinished_Below_Median_Executed Campaign 1.176471
198 SE Unfinished Below_Median_Executed SE_Unfinished_Below_Median_Executed Campaign 1.176471
202 RR Unfinished AboveMedianExecuted RR_Unfinished_AboveMedianExecuted Campaign 1.176471
203 RR Unfinished Below_Median_Executed RR_Unfinished_Below_Median_Executed Campaign 1.176471
204 MA Unfinished AboveMedianExecuted MA_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
208 MT Ongoing AboveMedianExecuted MT_Ongoing_AboveMedianExecuted Campaign 1.176471
210 AP Ongoing Below_Median_Executed AP_Ongoing_Below_Median_Executed Campaign 1.176471
228 MS Unfinished Below_Median_Executed MS_Unfinished_Below_Median_Executed ControlCampaign 6.666667
231 PA Unfinished AboveMedianExecuted PA_Unfinished_AboveMedianExecuted Campaign 1.176471
233 MG Ongoing AboveMedianExecuted MG_Ongoing_AboveMedianExecuted Campaign 1.176471
241 MS Unfinished AboveMedianExecuted MS_Unfinished_AboveMedianExecuted Campaign 1.176471
242 MT Stopped AboveMedianExecuted MT_Stopped_AboveMedianExecuted Campaign 1.176471
247 AM Unfinished AboveMedianExecuted AM_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
252 RO Unfinished AboveMedianExecuted RO_Unfinished_AboveMedianExecuted Campaign 1.176471
253 RO Ongoing Below_Median_Executed RO_Ongoing_Below_Median_Executed Campaign 1.176471
256 PI Ongoing AboveMedianExecuted PI_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
265 RN Ongoing AboveMedianExecuted RN_Ongoing_AboveMedianExecuted Campaign 1.176471
270 RN Ongoing AboveMedianExecuted RN_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
272 RN Stopped Below_Median_Executed RN_Stopped_Below_Median_Executed Campaign 1.176471
280 PE Unfinished AboveMedianExecuted PE_Unfinished_AboveMedianExecuted Campaign 1.176471
285 AL Ongoing AboveMedianExecuted AL_Ongoing_AboveMedianExecuted Campaign 1.176471
287 SE Unfinished AboveMedianExecuted SE_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
288 SE Ongoing AboveMedianExecuted SE_Ongoing_AboveMedianExecuted Campaign 1.176471
300 MG Ongoing Below_Median_Executed MG_Ongoing_Below_Median_Executed ControlCampaign 6.666667
313 SP Unfinished AboveMedianExecuted SP_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
316 RO Stopped AboveMedianExecuted RO_Stopped_AboveMedianExecuted Campaign 1.176471
319 PR Unfinished AboveMedianExecuted PR_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
320 PR Ongoing AboveMedianExecuted PR_Ongoing_AboveMedianExecuted Campaign 1.176471
321 PR Unfinished AboveMedianExecuted PR_Unfinished_AboveMedianExecuted Campaign 1.176471
327 RS Stopped AboveMedianExecuted RS_Stopped_AboveMedianExecuted ControlCampaign 6.666667
328 MT Ongoing Below_Median_Executed MT_Ongoing_Below_Median_Executed Campaign 1.176471
329 MT Stopped Below_Median_Executed MT_Stopped_Below_Median_Executed Campaign 1.176471
333 BA Ongoing Below_Median_Executed BA_Ongoing_Below_Median_Executed ControlCampaign 6.666667
340 BA Unfinished Below_Median_Executed BA_Unfinished_Below_Median_Executed ControlCampaign 6.666667
341 BA Unfinished AboveMedianExecuted BA_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
357 PR Stopped Below_Median_Executed PR_Stopped_Below_Median_Executed ControlCampaign 6.666667
359 PR Stopped Below_Median_Executed PR_Stopped_Below_Median_Executed Campaign 1.176471
371 RS Ongoing Below_Median_Executed RS_Ongoing_Below_Median_Executed Campaign 1.176471
384 SP Ongoing Below_Median_Executed SP_Ongoing_Below_Median_Executed ControlCampaign 6.666667
391 MG Ongoing AboveMedianExecuted MG_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
410 SE Unfinished AboveMedianExecuted SE_Unfinished_AboveMedianExecuted Campaign 1.176471
411 SE Ongoing Below_Median_Executed SE_Ongoing_Below_Median_Executed ControlCampaign 6.666667
415 AM Ongoing AboveMedianExecuted AM_Ongoing_AboveMedianExecuted Campaign 1.176471
418 BA Stopped AboveMedianExecuted BA_Stopped_AboveMedianExecuted Campaign 1.176471
420 SP Unfinished Below_Median_Executed SP_Unfinished_Below_Median_Executed Campaign 1.176471
422 SC Ongoing Below_Median_Executed SC_Ongoing_Below_Median_Executed ControlCampaign 6.666667
424 PE Ongoing AboveMedianExecuted PE_Ongoing_AboveMedianExecuted Campaign 1.176471
439 GO Unfinished Below_Median_Executed GO_Unfinished_Below_Median_Executed Campaign 1.176471
445 MS Ongoing Below_Median_Executed MS_Ongoing_Below_Median_Executed Campaign 1.176471
446 MS Ongoing AboveMedianExecuted MS_Ongoing_AboveMedianExecuted Campaign 1.176471
449 PB Ongoing Below_Median_Executed PB_Ongoing_Below_Median_Executed Campaign 1.176471
450 PE Unfinished Below_Median_Executed PE_Unfinished_Below_Median_Executed Campaign 1.176471
456 CE Stopped AboveMedianExecuted CE_Stopped_AboveMedianExecuted Campaign 1.176471
467 BA Unfinished Below_Median_Executed BA_Unfinished_Below_Median_Executed Campaign 1.176471
487 AL Unfinished Below_Median_Executed AL_Unfinished_Below_Median_Executed ControlCampaign 6.666667
488 BA Ongoing Below_Median_Executed BA_Ongoing_Below_Median_Executed Campaign 1.176471
496 GO Ongoing AboveMedianExecuted GO_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
505 MG Ongoing Below_Median_Executed MG_Ongoing_Below_Median_Executed Campaign 1.176471
507 MG Stopped AboveMedianExecuted MG_Stopped_AboveMedianExecuted ControlCampaign 6.666667
512 MT Ongoing AboveMedianExecuted MT_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
515 PB Unfinished AboveMedianExecuted PB_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
517 PE Ongoing AboveMedianExecuted PE_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
526 RN Stopped AboveMedianExecuted RN_Stopped_AboveMedianExecuted Campaign 1.176471
527 RS Ongoing Below_Median_Executed RS_Ongoing_Below_Median_Executed ControlCampaign 6.666667
530 RS Ongoing AboveMedianExecuted RS_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
531 SC Unfinished AboveMedianExecuted SC_Unfinished_AboveMedianExecuted Campaign 1.176471
537 AL Unfinished AboveMedianExecuted AL_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
559 SC Ongoing Below_Median_Executed SC_Ongoing_Below_Median_Executed Campaign 1.176471
560 SC Ongoing AboveMedianExecuted SC_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
561 SC Ongoing AboveMedianExecuted SC_Ongoing_AboveMedianExecuted Campaign 1.176471
562 SC Stopped AboveMedianExecuted SC_Stopped_AboveMedianExecuted ControlCampaign 6.666667
571 SP Ongoing Below_Median_Executed SP_Ongoing_Below_Median_Executed Campaign 1.176471
576 CE Ongoing Below_Median_Executed CE_Ongoing_Below_Median_Executed Campaign 1.176471
581 CE Ongoing AboveMedianExecuted CE_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
589 CE Unfinished Below_Median_Executed CE_Unfinished_Below_Median_Executed ControlCampaign 6.666667
604 PR Ongoing AboveMedianExecuted PR_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
625 MS Stopped AboveMedianExecuted MS_Stopped_AboveMedianExecuted Campaign 1.176471
627 MS Unfinished Below_Median_Executed MS_Unfinished_Below_Median_Executed Campaign 1.176471
637 PR Ongoing Below_Median_Executed PR_Ongoing_Below_Median_Executed Campaign 1.176471
658 PR Ongoing Below_Median_Executed PR_Ongoing_Below_Median_Executed ControlCampaign 6.666667
664 PI Unfinished Below_Median_Executed PI_Unfinished_Below_Median_Executed ControlCampaign 6.666667
668 RN Unfinished Below_Median_Executed RN_Unfinished_Below_Median_Executed Campaign 1.176471
671 MA Stopped AboveMedianExecuted MA_Stopped_AboveMedianExecuted ControlCampaign 6.666667
673 RO Unfinished Below_Median_Executed RO_Unfinished_Below_Median_Executed Campaign 1.176471
674 SE Ongoing Below_Median_Executed SE_Ongoing_Below_Median_Executed Campaign 1.176471
702 ES Unfinished AboveMedianExecuted ES_Unfinished_AboveMedianExecuted Campaign 1.176471
711 MT Unfinished Below_Median_Executed MT_Unfinished_Below_Median_Executed Campaign 1.176471
730 MT Stopped AboveMedianExecuted MT_Stopped_AboveMedianExecuted ControlCampaign 6.666667
734 PE Ongoing Below_Median_Executed PE_Ongoing_Below_Median_Executed Campaign 1.176471
735 PA Unfinished Below_Median_Executed PA_Unfinished_Below_Median_Executed ControlCampaign 6.666667
736 AL Ongoing Below_Median_Executed AL_Ongoing_Below_Median_Executed Campaign 1.176471
752 AM Stopped AboveMedianExecuted AM_Stopped_AboveMedianExecuted Campaign 1.176471
771 CE Stopped AboveMedianExecuted CE_Stopped_AboveMedianExecuted ControlCampaign 6.666667
782 ES Stopped AboveMedianExecuted ES_Stopped_AboveMedianExecuted Campaign 1.176471
783 RS Stopped AboveMedianExecuted RS_Stopped_AboveMedianExecuted Campaign 1.176471
786 AM Ongoing Below_Median_Executed AM_Ongoing_Below_Median_Executed ControlCampaign 6.666667
787 RS Stopped Below_Median_Executed RS_Stopped_Below_Median_Executed ControlCampaign 6.666667
791 PE Stopped AboveMedianExecuted PE_Stopped_AboveMedianExecuted Campaign 1.176471
801 RO Unfinished AboveMedianExecuted RO_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
803 AL Stopped AboveMedianExecuted AL_Stopped_AboveMedianExecuted Campaign 1.176471
804 SP Stopped AboveMedianExecuted SP_Stopped_AboveMedianExecuted ControlCampaign 6.666667
818 TO Stopped AboveMedianExecuted TO_Stopped_AboveMedianExecuted ControlCampaign 6.666667
820 SP Stopped AboveMedianExecuted SP_Stopped_AboveMedianExecuted Campaign 1.176471
830 AL Unfinished AboveMedianExecuted AL_Unfinished_AboveMedianExecuted Campaign 1.176471
834 RJ Ongoing Below_Median_Executed RJ_Ongoing_Below_Median_Executed ControlCampaign 6.666667
835 RJ Ongoing Below_Median_Executed RJ_Ongoing_Below_Median_Executed Campaign 1.176471
846 MA Ongoing AboveMedianExecuted MA_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
884 PB Stopped AboveMedianExecuted PB_Stopped_AboveMedianExecuted Campaign 1.176471
932 RO Ongoing AboveMedianExecuted RO_Ongoing_AboveMedianExecuted Campaign 1.176471
935 RJ Stopped AboveMedianExecuted RJ_Stopped_AboveMedianExecuted ControlCampaign 6.666667
936 RJ Stopped AboveMedianExecuted RJ_Stopped_AboveMedianExecuted Campaign 1.176471
949 PA Unfinished AboveMedianExecuted PA_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
952 RS Stopped Below_Median_Executed RS_Stopped_Below_Median_Executed Campaign 1.176471
989 RN Stopped AboveMedianExecuted RN_Stopped_AboveMedianExecuted ControlCampaign 6.666667
1006 TO Ongoing Below_Median_Executed TO_Ongoing_Below_Median_Executed Campaign 1.176471
1013 RJ Ongoing AboveMedianExecuted RJ_Ongoing_AboveMedianExecuted Campaign 1.176471
1033 PI Stopped AboveMedianExecuted PI_Stopped_AboveMedianExecuted Campaign 1.176471
1054 AL Stopped AboveMedianExecuted AL_Stopped_AboveMedianExecuted ControlCampaign 6.666667
1086 MA Stopped AboveMedianExecuted MA_Stopped_AboveMedianExecuted Campaign 1.176471
1102 BA Stopped Below_Median_Executed BA_Stopped_Below_Median_Executed ControlCampaign 6.666667
1118 BA Stopped Below_Median_Executed BA_Stopped_Below_Median_Executed Campaign 1.176471
1136 MS Stopped AboveMedianExecuted MS_Stopped_AboveMedianExecuted ControlCampaign 6.666667
1147 PE Stopped AboveMedianExecuted PE_Stopped_AboveMedianExecuted ControlCampaign 6.666667
1155 AC Ongoing AboveMedianExecuted AC_Ongoing_AboveMedianExecuted Campaign 1.176471
1158 AC Stopped AboveMedianExecuted AC_Stopped_AboveMedianExecuted Campaign 1.176471
1161 AL Unfinished Below_Median_Executed AL_Unfinished_Below_Median_Executed Campaign 1.176471
1163 AL Ongoing AboveMedianExecuted AL_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
1187 RN Ongoing Below_Median_Executed RN_Ongoing_Below_Median_Executed ControlCampaign 6.666667
1225 SP Stopped Below_Median_Executed SP_Stopped_Below_Median_Executed Campaign 1.176471
1231 SP Ongoing AboveMedianExecuted SP_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
1246 RJ Stopped Below_Median_Executed RJ_Stopped_Below_Median_Executed Campaign 1.176471
1264 ES Ongoing Below_Median_Executed ES_Ongoing_Below_Median_Executed ControlCampaign 6.666667
1279 SE Stopped AboveMedianExecuted SE_Stopped_AboveMedianExecuted Campaign 1.176471
1287 SE Stopped AboveMedianExecuted SE_Stopped_AboveMedianExecuted ControlCampaign 6.666667
1313 PI Ongoing Below_Median_Executed PI_Ongoing_Below_Median_Executed ControlCampaign 6.666667
1316 PI Ongoing Below_Median_Executed PI_Ongoing_Below_Median_Executed Campaign 1.176471
1318 PI Stopped AboveMedianExecuted PI_Stopped_AboveMedianExecuted ControlCampaign 6.666667
1338 RS Unfinished AboveMedianExecuted RS_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
1371 PE Unfinished Below_Median_Executed PE_Unfinished_Below_Median_Executed ControlCampaign 6.666667
1381 PE Ongoing Below_Median_Executed PE_Ongoing_Below_Median_Executed ControlCampaign 6.666667
1385 PA Unfinished Below_Median_Executed PA_Unfinished_Below_Median_Executed Campaign 1.176471
1402 PE Unfinished AboveMedianExecuted PE_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
1527 PB Ongoing AboveMedianExecuted PB_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
1560 AL Ongoing Below_Median_Executed AL_Ongoing_Below_Median_Executed ControlCampaign 6.666667
1639 SC Stopped AboveMedianExecuted SC_Stopped_AboveMedianExecuted Campaign 1.176471
1652 MA Stopped Below_Median_Executed MA_Stopped_Below_Median_Executed Campaign 1.176471
1670 MA Unfinished Below_Median_Executed MA_Unfinished_Below_Median_Executed ControlCampaign 6.666667
1771 AM Ongoing Below_Median_Executed AM_Ongoing_Below_Median_Executed Campaign 1.176471
1852 MS Ongoing AboveMedianExecuted MS_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
1948 TO Stopped Below_Median_Executed TO_Stopped_Below_Median_Executed ControlCampaign 6.666667
1952 SP Stopped Below_Median_Executed SP_Stopped_Below_Median_Executed ControlCampaign 6.666667
1963 RO Stopped AboveMedianExecuted RO_Stopped_AboveMedianExecuted ControlCampaign 6.666667
1971 RO Ongoing AboveMedianExecuted RO_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
1990 DF Ongoing Below_Median_Executed DF_Ongoing_Below_Median_Executed Campaign 1.176471
1995 AP Stopped AboveMedianExecuted AP_Stopped_AboveMedianExecuted Campaign 1.176471
2022 AC Ongoing AboveMedianExecuted AC_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
2026 AC Ongoing Below_Median_Executed AC_Ongoing_Below_Median_Executed Campaign 1.176471
2093 CE Stopped Below_Median_Executed CE_Stopped_Below_Median_Executed Campaign 1.176471
2108 DF Ongoing Below_Median_Executed DF_Ongoing_Below_Median_Executed ControlCampaign 6.666667
2206 RO Stopped Below_Median_Executed RO_Stopped_Below_Median_Executed Campaign 1.176471
2207 GO Stopped Below_Median_Executed GO_Stopped_Below_Median_Executed ControlCampaign 6.666667
2231 ES Ongoing AboveMedianExecuted ES_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
2271 TO Unfinished Below_Median_Executed TO_Unfinished_Below_Median_Executed ControlCampaign 6.666667
2274 AM Ongoing AboveMedianExecuted AM_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
2333 PA Stopped Below_Median_Executed PA_Stopped_Below_Median_Executed ControlCampaign 6.666667
2451 RS Unfinished Below_Median_Executed RS_Unfinished_Below_Median_Executed ControlCampaign 6.666667
2463 DF Ongoing AboveMedianExecuted DF_Ongoing_AboveMedianExecuted Campaign 1.176471
2467 RJ Unfinished AboveMedianExecuted RJ_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
2532 CE Ongoing Below_Median_Executed CE_Ongoing_Below_Median_Executed ControlCampaign 6.666667
2545 TO Ongoing Below_Median_Executed TO_Ongoing_Below_Median_Executed ControlCampaign 6.666667
2561 RN Ongoing Below_Median_Executed RN_Ongoing_Below_Median_Executed Campaign 1.176471
2570 PB Stopped Below_Median_Executed PB_Stopped_Below_Median_Executed ControlCampaign 6.666667
2571 PB Stopped Below_Median_Executed PB_Stopped_Below_Median_Executed Campaign 1.176471
2607 PB Ongoing Below_Median_Executed PB_Ongoing_Below_Median_Executed ControlCampaign 6.666667
2663 PE Stopped Below_Median_Executed PE_Stopped_Below_Median_Executed Campaign 1.176471
2695 MS Stopped Below_Median_Executed MS_Stopped_Below_Median_Executed Campaign 1.176471
2736 PI Stopped Below_Median_Executed PI_Stopped_Below_Median_Executed Campaign 1.176471
2791 RO Ongoing Below_Median_Executed RO_Ongoing_Below_Median_Executed ControlCampaign 6.666667
2847 MT Ongoing Below_Median_Executed MT_Ongoing_Below_Median_Executed ControlCampaign 6.666667
2862 RJ Ongoing AboveMedianExecuted RJ_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
2874 AC Stopped Below_Median_Executed AC_Stopped_Below_Median_Executed Campaign 1.176471
2995 SP Unfinished Below_Median_Executed SP_Unfinished_Below_Median_Executed ControlCampaign 6.666667
3117 TO Stopped Below_Median_Executed TO_Stopped_Below_Median_Executed Campaign 1.176471
3262 SC Unfinished Below_Median_Executed SC_Unfinished_Below_Median_Executed Campaign 1.176471
3301 MT Unfinished Below_Median_Executed MT_Unfinished_Below_Median_Executed ControlCampaign 6.666667
3362 PR Unfinished Below_Median_Executed PR_Unfinished_Below_Median_Executed ControlCampaign 6.666667
3485 RR Ongoing Below_Median_Executed RR_Ongoing_Below_Median_Executed ControlCampaign 6.666667
3488 RR Stopped AboveMedianExecuted RR_Stopped_AboveMedianExecuted ControlCampaign 6.666667
3511 ES Stopped Below_Median_Executed ES_Stopped_Below_Median_Executed Campaign 1.176471
3533 AP Ongoing AboveMedianExecuted AP_Ongoing_AboveMedianExecuted Campaign 1.176471
3574 MA Stopped Below_Median_Executed MA_Stopped_Below_Median_Executed ControlCampaign 6.666667
3846 SC Stopped Below_Median_Executed SC_Stopped_Below_Median_Executed Campaign 1.176471
3866 TO Ongoing AboveMedianExecuted TO_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
3909 ES Unfinished Below_Median_Executed ES_Unfinished_Below_Median_Executed ControlCampaign 6.666667
3971 AP Ongoing Below_Median_Executed AP_Ongoing_Below_Median_Executed ControlCampaign 6.666667
4167 MG Stopped Below_Median_Executed MG_Stopped_Below_Median_Executed Campaign 1.176471
4527 PE Stopped Below_Median_Executed PE_Stopped_Below_Median_Executed ControlCampaign 6.666667
4571 MT Stopped Below_Median_Executed MT_Stopped_Below_Median_Executed ControlCampaign 6.666667
4656 ES Unfinished Below_Median_Executed ES_Unfinished_Below_Median_Executed Campaign 1.176471
4738 PB Stopped AboveMedianExecuted PB_Stopped_AboveMedianExecuted ControlCampaign 6.666667
4807 CE Stopped Below_Median_Executed CE_Stopped_Below_Median_Executed ControlCampaign 6.666667
4844 AP Stopped Below_Median_Executed AP_Stopped_Below_Median_Executed Campaign 1.176471
4989 AC Ongoing Below_Median_Executed AC_Ongoing_Below_Median_Executed ControlCampaign 6.666667
5159 AL Stopped Below_Median_Executed AL_Stopped_Below_Median_Executed Campaign 1.176471
5471 AC Stopped AboveMedianExecuted AC_Stopped_AboveMedianExecuted ControlCampaign 6.666667
6030 RR Unfinished Below_Median_Executed RR_Unfinished_Below_Median_Executed ControlCampaign 6.666667
6087 RO Unfinished Below_Median_Executed RO_Unfinished_Below_Median_Executed ControlCampaign 6.666667
6095 SE Ongoing AboveMedianExecuted SE_Ongoing_AboveMedianExecuted ControlCampaign 6.666667
6118 PI Stopped Below_Median_Executed PI_Stopped_Below_Median_Executed ControlCampaign 6.666667
6346 MS Ongoing Below_Median_Executed MS_Ongoing_Below_Median_Executed ControlCampaign 6.666667
6400 AP Stopped AboveMedianExecuted AP_Stopped_AboveMedianExecuted ControlCampaign 6.666667
6458 RR Unfinished AboveMedianExecuted RR_Unfinished_AboveMedianExecuted ControlCampaign 6.666667
6631 AL Stopped Below_Median_Executed AL_Stopped_Below_Median_Executed ControlCampaign 6.666667
6992 RJ Stopped Below_Median_Executed RJ_Stopped_Below_Median_Executed ControlCampaign 6.666667
Block randomization – Intervention 3
state status above_median_executed var_blocking Z_intheapp IPW_intheapp
8 SP Ongoing AboveMedianExecuted SP_Ongoing_AboveMedianExecuted OnApp 1.176471
11 MG Ongoing Below_Median_Executed MG_Ongoing_Below_Median_Executed OnApp 1.176471
12 SP Ongoing Below_Median_Executed SP_Ongoing_Below_Median_Executed OffApp 6.666667
22 RO Ongoing Below_Median_Executed RO_Ongoing_Below_Median_Executed OnApp 1.176471
51 ES Ongoing Below_Median_Executed ES_Ongoing_Below_Median_Executed OnApp 1.176471
102 BA Unfinished AboveMedianExecuted BA_Unfinished_AboveMedianExecuted OnApp 1.176471
131 TO Ongoing AboveMedianExecuted TO_Ongoing_AboveMedianExecuted OnApp 1.176471
176 MA Ongoing Below_Median_Executed MA_Ongoing_Below_Median_Executed OffApp 6.666667
198 SE Unfinished Below_Median_Executed SE_Unfinished_Below_Median_Executed OnApp 1.176471
199 PI Unfinished Below_Median_Executed PI_Unfinished_Below_Median_Executed OffApp 6.666667
200 PI Unfinished Below_Median_Executed PI_Unfinished_Below_Median_Executed OnApp 1.176471
201 MG Unfinished AboveMedianExecuted MG_Unfinished_AboveMedianExecuted OnApp 1.176471
233 MG Ongoing AboveMedianExecuted MG_Ongoing_AboveMedianExecuted OnApp 1.176471
270 RN Ongoing AboveMedianExecuted RN_Ongoing_AboveMedianExecuted OnApp 1.176471
272 RN Stopped Below_Median_Executed RN_Stopped_Below_Median_Executed OnApp 1.176471
335 MA Unfinished AboveMedianExecuted MA_Unfinished_AboveMedianExecuted OnApp 1.176471
338 CE Unfinished AboveMedianExecuted CE_Unfinished_AboveMedianExecuted OnApp 1.176471
339 SE Ongoing AboveMedianExecuted SE_Ongoing_AboveMedianExecuted OnApp 1.176471
340 BA Unfinished Below_Median_Executed BA_Unfinished_Below_Median_Executed OnApp 1.176471
374 MA Ongoing AboveMedianExecuted MA_Ongoing_AboveMedianExecuted OnApp 1.176471
377 MA Ongoing Below_Median_Executed MA_Ongoing_Below_Median_Executed OnApp 1.176471
397 PA Unfinished AboveMedianExecuted PA_Unfinished_AboveMedianExecuted OnApp 1.176471
401 MS Ongoing AboveMedianExecuted MS_Ongoing_AboveMedianExecuted OffApp 6.666667
419 SP Ongoing Below_Median_Executed SP_Ongoing_Below_Median_Executed OnApp 1.176471
445 MS Ongoing Below_Median_Executed MS_Ongoing_Below_Median_Executed OnApp 1.176471
449 PB Ongoing Below_Median_Executed PB_Ongoing_Below_Median_Executed OnApp 1.176471
456 CE Stopped AboveMedianExecuted CE_Stopped_AboveMedianExecuted OnApp 1.176471
459 AL Ongoing AboveMedianExecuted AL_Ongoing_AboveMedianExecuted OnApp 1.176471
460 BA Unfinished AboveMedianExecuted BA_Unfinished_AboveMedianExecuted OffApp 6.666667
511 MT Ongoing Below_Median_Executed MT_Ongoing_Below_Median_Executed OnApp 1.176471
537 AL Unfinished AboveMedianExecuted AL_Unfinished_AboveMedianExecuted OnApp 1.176471
539 MA Unfinished AboveMedianExecuted MA_Unfinished_AboveMedianExecuted OffApp 6.666667
542 MG Stopped AboveMedianExecuted MG_Stopped_AboveMedianExecuted OnApp 1.176471
543 PE Unfinished Below_Median_Executed PE_Unfinished_Below_Median_Executed OnApp 1.176471
545 TO Stopped AboveMedianExecuted TO_Stopped_AboveMedianExecuted OnApp 1.176471
558 RS Ongoing AboveMedianExecuted RS_Ongoing_AboveMedianExecuted OnApp 1.176471
559 SC Ongoing Below_Median_Executed SC_Ongoing_Below_Median_Executed OnApp 1.176471
562 SC Stopped AboveMedianExecuted SC_Stopped_AboveMedianExecuted OnApp 1.176471
564 AM Unfinished AboveMedianExecuted AM_Unfinished_AboveMedianExecuted OnApp 1.176471
573 SP Unfinished AboveMedianExecuted SP_Unfinished_AboveMedianExecuted OnApp 1.176471
580 TO Ongoing AboveMedianExecuted TO_Ongoing_AboveMedianExecuted OffApp 6.666667
591 TO Unfinished AboveMedianExecuted TO_Unfinished_AboveMedianExecuted OnApp 1.176471
592 GO Unfinished AboveMedianExecuted GO_Unfinished_AboveMedianExecuted OnApp 1.176471
593 GO Unfinished AboveMedianExecuted GO_Unfinished_AboveMedianExecuted OffApp 6.666667
597 SC Ongoing AboveMedianExecuted SC_Ongoing_AboveMedianExecuted OnApp 1.176471
605 RN Unfinished AboveMedianExecuted RN_Unfinished_AboveMedianExecuted OnApp 1.176471
624 MS Unfinished AboveMedianExecuted MS_Unfinished_AboveMedianExecuted OffApp 6.666667
626 MS Unfinished AboveMedianExecuted MS_Unfinished_AboveMedianExecuted OnApp 1.176471
627 MS Unfinished Below_Median_Executed MS_Unfinished_Below_Median_Executed OnApp 1.176471
631 MT Ongoing AboveMedianExecuted MT_Ongoing_AboveMedianExecuted OnApp 1.176471
632 MT Ongoing AboveMedianExecuted MT_Ongoing_AboveMedianExecuted OffApp 6.666667
640 SE Unfinished AboveMedianExecuted SE_Unfinished_AboveMedianExecuted OnApp 1.176471
645 ES Ongoing AboveMedianExecuted ES_Ongoing_AboveMedianExecuted OnApp 1.176471
674 SE Ongoing Below_Median_Executed SE_Ongoing_Below_Median_Executed OnApp 1.176471
676 RS Ongoing Below_Median_Executed RS_Ongoing_Below_Median_Executed OnApp 1.176471
700 MS Ongoing AboveMedianExecuted MS_Ongoing_AboveMedianExecuted OnApp 1.176471
709 MG Unfinished Below_Median_Executed MG_Unfinished_Below_Median_Executed OnApp 1.176471
710 PR Ongoing Below_Median_Executed PR_Ongoing_Below_Median_Executed OnApp 1.176471
770 RO Stopped AboveMedianExecuted RO_Stopped_AboveMedianExecuted OnApp 1.176471
776 PE Ongoing Below_Median_Executed PE_Ongoing_Below_Median_Executed OnApp 1.176471
808 BA Ongoing Below_Median_Executed BA_Ongoing_Below_Median_Executed OffApp 6.666667
825 GO Ongoing Below_Median_Executed GO_Ongoing_Below_Median_Executed OnApp 1.176471
861 AL Ongoing Below_Median_Executed AL_Ongoing_Below_Median_Executed OnApp 1.176471
868 TO Ongoing Below_Median_Executed TO_Ongoing_Below_Median_Executed OnApp 1.176471
870 GO Ongoing AboveMedianExecuted GO_Ongoing_AboveMedianExecuted OffApp 6.666667
899 PB Ongoing Below_Median_Executed PB_Ongoing_Below_Median_Executed OffApp 6.666667
913 RS Ongoing Below_Median_Executed RS_Ongoing_Below_Median_Executed OffApp 6.666667
932 RO Ongoing AboveMedianExecuted RO_Ongoing_AboveMedianExecuted OffApp 6.666667
941 ES Ongoing AboveMedianExecuted ES_Ongoing_AboveMedianExecuted OffApp 6.666667
958 CE Ongoing AboveMedianExecuted CE_Ongoing_AboveMedianExecuted OnApp 1.176471
973 PE Ongoing AboveMedianExecuted PE_Ongoing_AboveMedianExecuted OnApp 1.176471
999 GO Ongoing AboveMedianExecuted GO_Ongoing_AboveMedianExecuted OnApp 1.176471
1001 PE Stopped Below_Median_Executed PE_Stopped_Below_Median_Executed OnApp 1.176471
1008 AL Stopped AboveMedianExecuted AL_Stopped_AboveMedianExecuted OnApp 1.176471
1060 MG Ongoing AboveMedianExecuted MG_Ongoing_AboveMedianExecuted OffApp 6.666667
1064 BA Ongoing Below_Median_Executed BA_Ongoing_Below_Median_Executed OnApp 1.176471
1077 RN Ongoing Below_Median_Executed RN_Ongoing_Below_Median_Executed OnApp 1.176471
1085 RJ Ongoing Below_Median_Executed RJ_Ongoing_Below_Median_Executed OffApp 6.666667
1091 PB Ongoing AboveMedianExecuted PB_Ongoing_AboveMedianExecuted OnApp 1.176471
1128 RO Ongoing AboveMedianExecuted RO_Ongoing_AboveMedianExecuted OnApp 1.176471
1136 MS Stopped AboveMedianExecuted MS_Stopped_AboveMedianExecuted OnApp 1.176471
1157 AC Ongoing Below_Median_Executed AC_Ongoing_Below_Median_Executed OnApp 1.176471
1182 MG Ongoing Below_Median_Executed MG_Ongoing_Below_Median_Executed OffApp 6.666667
1184 PR Ongoing AboveMedianExecuted PR_Ongoing_AboveMedianExecuted OnApp 1.176471
1209 RS Unfinished Below_Median_Executed RS_Unfinished_Below_Median_Executed OnApp 1.176471
1225 SP Stopped Below_Median_Executed SP_Stopped_Below_Median_Executed OnApp 1.176471
1236 AC Ongoing AboveMedianExecuted AC_Ongoing_AboveMedianExecuted OnApp 1.176471
1247 RJ Ongoing Below_Median_Executed RJ_Ongoing_Below_Median_Executed OnApp 1.176471
1253 RJ Ongoing AboveMedianExecuted RJ_Ongoing_AboveMedianExecuted OnApp 1.176471
1284 SE Ongoing Below_Median_Executed SE_Ongoing_Below_Median_Executed OffApp 6.666667
1296 PI Ongoing AboveMedianExecuted PI_Ongoing_AboveMedianExecuted OnApp 1.176471
1313 PI Ongoing Below_Median_Executed PI_Ongoing_Below_Median_Executed OffApp 6.666667
1319 PI Ongoing Below_Median_Executed PI_Ongoing_Below_Median_Executed OnApp 1.176471
1320 PI Ongoing AboveMedianExecuted PI_Ongoing_AboveMedianExecuted OffApp 6.666667
1337 RS Unfinished AboveMedianExecuted RS_Unfinished_AboveMedianExecuted OnApp 1.176471
1375 PE Ongoing Below_Median_Executed PE_Ongoing_Below_Median_Executed OffApp 6.666667
1379 PA Ongoing Below_Median_Executed PA_Ongoing_Below_Median_Executed OnApp 1.176471
1463 GO Stopped AboveMedianExecuted GO_Stopped_AboveMedianExecuted OnApp 1.176471
1464 GO Stopped AboveMedianExecuted GO_Stopped_AboveMedianExecuted OffApp 6.666667
1508 RO Stopped AboveMedianExecuted RO_Stopped_AboveMedianExecuted OffApp 6.666667
1516 RO Ongoing Below_Median_Executed RO_Ongoing_Below_Median_Executed OffApp 6.666667
1519 ES Ongoing Below_Median_Executed ES_Ongoing_Below_Median_Executed OffApp 6.666667
1567 CE Ongoing Below_Median_Executed CE_Ongoing_Below_Median_Executed OffApp 6.666667
1568 CE Ongoing Below_Median_Executed CE_Ongoing_Below_Median_Executed OnApp 1.176471
1589 BA Ongoing AboveMedianExecuted BA_Ongoing_AboveMedianExecuted OnApp 1.176471
1614 BA Stopped Below_Median_Executed BA_Stopped_Below_Median_Executed OnApp 1.176471
1638 SC Ongoing Below_Median_Executed SC_Ongoing_Below_Median_Executed OffApp 6.666667
1739 PR Ongoing Below_Median_Executed PR_Ongoing_Below_Median_Executed OffApp 6.666667
1777 AM Ongoing AboveMedianExecuted AM_Ongoing_AboveMedianExecuted OnApp 1.176471
1799 AM Ongoing Below_Median_Executed AM_Ongoing_Below_Median_Executed OnApp 1.176471
1817 AM Stopped Below_Median_Executed AM_Stopped_Below_Median_Executed OnApp 1.176471
1851 MS Ongoing Below_Median_Executed MS_Ongoing_Below_Median_Executed OffApp 6.666667
1856 TO Ongoing Below_Median_Executed TO_Ongoing_Below_Median_Executed OffApp 6.666667
1952 SP Stopped Below_Median_Executed SP_Stopped_Below_Median_Executed OffApp 6.666667
1953 SP Stopped AboveMedianExecuted SP_Stopped_AboveMedianExecuted OnApp 1.176471
1992 BA Ongoing AboveMedianExecuted BA_Ongoing_AboveMedianExecuted OffApp 6.666667
1995 AP Stopped AboveMedianExecuted AP_Stopped_AboveMedianExecuted OnApp 1.176471
1996 AP Stopped AboveMedianExecuted AP_Stopped_AboveMedianExecuted OffApp 6.666667
1997 RO Unfinished Below_Median_Executed RO_Unfinished_Below_Median_Executed OffApp 6.666667
1998 PI Stopped AboveMedianExecuted PI_Stopped_AboveMedianExecuted OnApp 1.176471
2003 RN Stopped AboveMedianExecuted RN_Stopped_AboveMedianExecuted OnApp 1.176471
2129 MT Unfinished Below_Median_Executed MT_Unfinished_Below_Median_Executed OnApp 1.176471
2130 MT Unfinished AboveMedianExecuted MT_Unfinished_AboveMedianExecuted OffApp 6.666667
2145 PB Unfinished AboveMedianExecuted PB_Unfinished_AboveMedianExecuted OnApp 1.176471
2151 RO Unfinished AboveMedianExecuted RO_Unfinished_AboveMedianExecuted OnApp 1.176471
2172 CE Ongoing AboveMedianExecuted CE_Ongoing_AboveMedianExecuted OffApp 6.666667
2243 MA Ongoing AboveMedianExecuted MA_Ongoing_AboveMedianExecuted OffApp 6.666667
2248 PA Ongoing AboveMedianExecuted PA_Ongoing_AboveMedianExecuted OnApp 1.176471
2251 PA Ongoing AboveMedianExecuted PA_Ongoing_AboveMedianExecuted OffApp 6.666667
2255 PE Stopped AboveMedianExecuted PE_Stopped_AboveMedianExecuted OnApp 1.176471
2256 PB Unfinished AboveMedianExecuted PB_Unfinished_AboveMedianExecuted OffApp 6.666667
2258 PE Unfinished Below_Median_Executed PE_Unfinished_Below_Median_Executed OffApp 6.666667
2262 PI Unfinished AboveMedianExecuted PI_Unfinished_AboveMedianExecuted OnApp 1.176471
2270 TO Unfinished Below_Median_Executed TO_Unfinished_Below_Median_Executed OnApp 1.176471
2272 AM Unfinished Below_Median_Executed AM_Unfinished_Below_Median_Executed OnApp 1.176471
2281 MT Stopped AboveMedianExecuted MT_Stopped_AboveMedianExecuted OnApp 1.176471
2288 MT Stopped Below_Median_Executed MT_Stopped_Below_Median_Executed OnApp 1.176471
2294 AM Unfinished Below_Median_Executed AM_Unfinished_Below_Median_Executed OffApp 6.666667
2297 AM Ongoing AboveMedianExecuted AM_Ongoing_AboveMedianExecuted OffApp 6.666667
2303 AM Ongoing Below_Median_Executed AM_Ongoing_Below_Median_Executed OffApp 6.666667
2310 MA Unfinished Below_Median_Executed MA_Unfinished_Below_Median_Executed OnApp 1.176471
2333 PA Stopped Below_Median_Executed PA_Stopped_Below_Median_Executed OnApp 1.176471
2335 PA Stopped AboveMedianExecuted PA_Stopped_AboveMedianExecuted OnApp 1.176471
2384 SP Ongoing AboveMedianExecuted SP_Ongoing_AboveMedianExecuted OffApp 6.666667
2418 GO Stopped Below_Median_Executed GO_Stopped_Below_Median_Executed OnApp 1.176471
2433 SC Ongoing AboveMedianExecuted SC_Ongoing_AboveMedianExecuted OffApp 6.666667
2559 RN Ongoing Below_Median_Executed RN_Ongoing_Below_Median_Executed OffApp 6.666667
2632 GO Ongoing Below_Median_Executed GO_Ongoing_Below_Median_Executed OffApp 6.666667
2665 MA Stopped AboveMedianExecuted MA_Stopped_AboveMedianExecuted OffApp 6.666667
2676 PR Ongoing AboveMedianExecuted PR_Ongoing_AboveMedianExecuted OffApp 6.666667
2695 MS Stopped Below_Median_Executed MS_Stopped_Below_Median_Executed OnApp 1.176471
2698 AL Ongoing AboveMedianExecuted AL_Ongoing_AboveMedianExecuted OffApp 6.666667
2736 PI Stopped Below_Median_Executed PI_Stopped_Below_Median_Executed OnApp 1.176471
2743 MA Stopped Below_Median_Executed MA_Stopped_Below_Median_Executed OffApp 6.666667
2769 MA Unfinished Below_Median_Executed MA_Unfinished_Below_Median_Executed OffApp 6.666667
2771 MA Stopped Below_Median_Executed MA_Stopped_Below_Median_Executed OnApp 1.176471
2790 AL Ongoing Below_Median_Executed AL_Ongoing_Below_Median_Executed OffApp 6.666667
2797 AL Stopped Below_Median_Executed AL_Stopped_Below_Median_Executed OnApp 1.176471
2852 PR Stopped AboveMedianExecuted PR_Stopped_AboveMedianExecuted OnApp 1.176471
2855 PE Ongoing AboveMedianExecuted PE_Ongoing_AboveMedianExecuted OffApp 6.666667
2862 RJ Ongoing AboveMedianExecuted RJ_Ongoing_AboveMedianExecuted OffApp 6.666667
2868 AC Stopped AboveMedianExecuted AC_Stopped_AboveMedianExecuted OnApp 1.176471
2871 AC Ongoing AboveMedianExecuted AC_Ongoing_AboveMedianExecuted OffApp 6.666667
2874 AC Stopped Below_Median_Executed AC_Stopped_Below_Median_Executed OnApp 1.176471
2877 BA Stopped AboveMedianExecuted BA_Stopped_AboveMedianExecuted OffApp 6.666667
2901 MS Stopped AboveMedianExecuted MS_Stopped_AboveMedianExecuted OffApp 6.666667
2942 BA Stopped Below_Median_Executed BA_Stopped_Below_Median_Executed OffApp 6.666667
2995 SP Unfinished Below_Median_Executed SP_Unfinished_Below_Median_Executed OnApp 1.176471
3069 MA Stopped AboveMedianExecuted MA_Stopped_AboveMedianExecuted OnApp 1.176471
3079 BA Stopped AboveMedianExecuted BA_Stopped_AboveMedianExecuted OnApp 1.176471
3110 RR Ongoing Below_Median_Executed RR_Ongoing_Below_Median_Executed OnApp 1.176471
3116 RR Unfinished AboveMedianExecuted RR_Unfinished_AboveMedianExecuted OffApp 6.666667
3117 TO Stopped Below_Median_Executed TO_Stopped_Below_Median_Executed OnApp 1.176471
3128 PA Unfinished AboveMedianExecuted PA_Unfinished_AboveMedianExecuted OffApp 6.666667
3137 PA Ongoing Below_Median_Executed PA_Ongoing_Below_Median_Executed OffApp 6.666667
3262 SC Unfinished Below_Median_Executed SC_Unfinished_Below_Median_Executed OffApp 6.666667
3278 AC Ongoing Below_Median_Executed AC_Ongoing_Below_Median_Executed OffApp 6.666667
3293 PR Stopped Below_Median_Executed PR_Stopped_Below_Median_Executed OnApp 1.176471
3301 MT Unfinished Below_Median_Executed MT_Unfinished_Below_Median_Executed OffApp 6.666667
3360 PR Unfinished Below_Median_Executed PR_Unfinished_Below_Median_Executed OnApp 1.176471
3390 CE Stopped AboveMedianExecuted CE_Stopped_AboveMedianExecuted OffApp 6.666667
3397 RJ Stopped Below_Median_Executed RJ_Stopped_Below_Median_Executed OnApp 1.176471
3468 ES Stopped AboveMedianExecuted ES_Stopped_AboveMedianExecuted OnApp 1.176471
3484 RR Stopped AboveMedianExecuted RR_Stopped_AboveMedianExecuted OffApp 6.666667
3487 RR Stopped AboveMedianExecuted RR_Stopped_AboveMedianExecuted OnApp 1.176471
3533 AP Ongoing AboveMedianExecuted AP_Ongoing_AboveMedianExecuted OnApp 1.176471
3534 AP Ongoing Below_Median_Executed AP_Ongoing_Below_Median_Executed OnApp 1.176471
3613 CE Unfinished Below_Median_Executed CE_Unfinished_Below_Median_Executed OnApp 1.176471
3644 RS Ongoing AboveMedianExecuted RS_Ongoing_AboveMedianExecuted OffApp 6.666667
3660 PR Stopped AboveMedianExecuted PR_Stopped_AboveMedianExecuted OffApp 6.666667
3691 AM Stopped AboveMedianExecuted AM_Stopped_AboveMedianExecuted OnApp 1.176471
3760 PA Unfinished Below_Median_Executed PA_Unfinished_Below_Median_Executed OnApp 1.176471
3801 SC Stopped AboveMedianExecuted SC_Stopped_AboveMedianExecuted OffApp 6.666667
3842 RR Unfinished Below_Median_Executed RR_Unfinished_Below_Median_Executed OnApp 1.176471
3846 SC Stopped Below_Median_Executed SC_Stopped_Below_Median_Executed OnApp 1.176471
3909 ES Unfinished Below_Median_Executed ES_Unfinished_Below_Median_Executed OnApp 1.176471
3993 PB Stopped Below_Median_Executed PB_Stopped_Below_Median_Executed OffApp 6.666667
4006 AM Stopped AboveMedianExecuted AM_Stopped_AboveMedianExecuted OffApp 6.666667
4011 CE Stopped Below_Median_Executed CE_Stopped_Below_Median_Executed OnApp 1.176471
4040 AP Ongoing Below_Median_Executed AP_Ongoing_Below_Median_Executed OffApp 6.666667
4103 MG Stopped AboveMedianExecuted MG_Stopped_AboveMedianExecuted OffApp 6.666667
4167 MG Stopped Below_Median_Executed MG_Stopped_Below_Median_Executed OnApp 1.176471
4390 PA Stopped Below_Median_Executed PA_Stopped_Below_Median_Executed OffApp 6.666667
4455 RO Stopped Below_Median_Executed RO_Stopped_Below_Median_Executed OnApp 1.176471
4494 PA Unfinished Below_Median_Executed PA_Unfinished_Below_Median_Executed OffApp 6.666667
4507 RS Stopped Below_Median_Executed RS_Stopped_Below_Median_Executed OnApp 1.176471
4511 PB Stopped AboveMedianExecuted PB_Stopped_AboveMedianExecuted OnApp 1.176471
4540 TO Stopped AboveMedianExecuted TO_Stopped_AboveMedianExecuted OffApp 6.666667
4571 MT Stopped Below_Median_Executed MT_Stopped_Below_Median_Executed OffApp 6.666667
4580 PA Stopped AboveMedianExecuted PA_Stopped_AboveMedianExecuted OffApp 6.666667
4661 AL Unfinished AboveMedianExecuted AL_Unfinished_AboveMedianExecuted OffApp 6.666667
4672 ES Stopped Below_Median_Executed ES_Stopped_Below_Median_Executed OnApp 1.176471
4696 MT Unfinished AboveMedianExecuted MT_Unfinished_AboveMedianExecuted OnApp 1.176471
4718 RS Stopped AboveMedianExecuted RS_Stopped_AboveMedianExecuted OnApp 1.176471
4806 AM Unfinished AboveMedianExecuted AM_Unfinished_AboveMedianExecuted OffApp 6.666667
4844 AP Stopped Below_Median_Executed AP_Stopped_Below_Median_Executed OnApp 1.176471
4846 AP Unfinished Below_Median_Executed AP_Unfinished_Below_Median_Executed OnApp 1.176471
4915 PI Unfinished AboveMedianExecuted PI_Unfinished_AboveMedianExecuted OffApp 6.666667
4934 PI Stopped AboveMedianExecuted PI_Stopped_AboveMedianExecuted OffApp 6.666667
4940 RJ Unfinished Below_Median_Executed RJ_Unfinished_Below_Median_Executed OnApp 1.176471
4947 GO Unfinished Below_Median_Executed GO_Unfinished_Below_Median_Executed OnApp 1.176471
4960 PR Unfinished AboveMedianExecuted PR_Unfinished_AboveMedianExecuted OnApp 1.176471
5040 CE Stopped Below_Median_Executed CE_Stopped_Below_Median_Executed OffApp 6.666667
5115 TO Unfinished AboveMedianExecuted TO_Unfinished_AboveMedianExecuted OffApp 6.666667
5329 RR Unfinished AboveMedianExecuted RR_Unfinished_AboveMedianExecuted OnApp 1.176471
5440 RN Ongoing AboveMedianExecuted RN_Ongoing_AboveMedianExecuted OffApp 6.666667
5471 AC Stopped AboveMedianExecuted AC_Stopped_AboveMedianExecuted OffApp 6.666667
5531 AP Ongoing AboveMedianExecuted AP_Ongoing_AboveMedianExecuted OffApp 6.666667
5565 MT Ongoing Below_Median_Executed MT_Ongoing_Below_Median_Executed OffApp 6.666667
5571 PB Unfinished Below_Median_Executed PB_Unfinished_Below_Median_Executed OnApp 1.176471
5573 PB Ongoing AboveMedianExecuted PB_Ongoing_AboveMedianExecuted OffApp 6.666667
5581 PI Stopped Below_Median_Executed PI_Stopped_Below_Median_Executed OffApp 6.666667
5600 PR Stopped Below_Median_Executed PR_Stopped_Below_Median_Executed OffApp 6.666667
5613 SE Ongoing AboveMedianExecuted SE_Ongoing_AboveMedianExecuted OffApp 6.666667
5620 RS Unfinished Below_Median_Executed RS_Unfinished_Below_Median_Executed OffApp 6.666667
5709 RR Ongoing Below_Median_Executed RR_Ongoing_Below_Median_Executed OffApp 6.666667
5791 BA Unfinished Below_Median_Executed BA_Unfinished_Below_Median_Executed OffApp 6.666667
5859 PE Unfinished AboveMedianExecuted PE_Unfinished_AboveMedianExecuted OnApp 1.176471
5890 PE Stopped AboveMedianExecuted PE_Stopped_AboveMedianExecuted OffApp 6.666667
5995 CE Unfinished AboveMedianExecuted CE_Unfinished_AboveMedianExecuted OffApp 6.666667
6012 MT Stopped AboveMedianExecuted MT_Stopped_AboveMedianExecuted OffApp 6.666667
6044 PE Stopped Below_Median_Executed PE_Stopped_Below_Median_Executed OffApp 6.666667
6049 RR Ongoing AboveMedianExecuted RR_Ongoing_AboveMedianExecuted OnApp 1.176471
6087 RO Unfinished Below_Median_Executed RO_Unfinished_Below_Median_Executed OnApp 1.176471
6146 PB Stopped Below_Median_Executed PB_Stopped_Below_Median_Executed OnApp 1.176471
6401 AP Stopped Below_Median_Executed AP_Stopped_Below_Median_Executed OffApp 6.666667
6459 RR Unfinished Below_Median_Executed RR_Unfinished_Below_Median_Executed OffApp 6.666667
6514 MG Stopped Below_Median_Executed MG_Stopped_Below_Median_Executed OffApp 6.666667
6532 AL Stopped Below_Median_Executed AL_Stopped_Below_Median_Executed OffApp 6.666667
6855 PB Stopped AboveMedianExecuted PB_Stopped_AboveMedianExecuted OffApp 6.666667
6862 PR Unfinished AboveMedianExecuted PR_Unfinished_AboveMedianExecuted OffApp 6.666667
6917 AL Unfinished Below_Median_Executed AL_Unfinished_Below_Median_Executed OnApp 1.176471
6948 SC Unfinished AboveMedianExecuted SC_Unfinished_AboveMedianExecuted OnApp 1.176471
7164 RO Unfinished AboveMedianExecuted RO_Unfinished_AboveMedianExecuted OffApp 6.666667

As pre-treatment variables, we used the municipal-level characteristics below:

  1. Log of Municipal Population in 2015
  2. Log of Number of Poor Families (2010 IBGE Census)
  3. Log of Total Federal Transfers to the Municipality in 2016
  4. IDEB Indicator for primary school quality (2015 Ministry of Education)
  5. IDEB Indicator for secondary school quality (2015 Ministry of Education)

All the pre-treatment variables were not significant, as displayed in the main text.

D. Treatments

  • Treatment groups:
    • Intervention 1: Municipality with all schools funded by the Ministry of Education showing up in the App.
    • Intervention 2: School construction showing up in the App.
    • Intervention 3: Mayor’s office received an email from the Ta de Pe bot requesting information about why the construction is delayed.
  • Control groups:
    • Intervention 1: Municipality with all schools funded by the Ministry of Education not showing up in the App.
    • Intervention 2: Selected school constructions not showing up in the App.
    • Intervention 3: Mayor’s office not receiving an email from the Ta de Pe bot.
  • Method of manipulation delivery:
    • Intervention 1: The municipalities in the treatment had all their school constructions showing up in the App.
    • Intervention 2: The schools in the treatment group were showing up in the App.
    • Intervention 3: The mayor’s office in the treatment group received a message from the Ta de Pe bot, requesting information about the school construction.
  • Software: The TDP App is an Android and iOS Application developed to facilitate bottom-up pressure on school constructions in Brazil.

E. Results

  • Outcome measures: We use six outcome measures, all taken from the Ministry of Education biannual report:
  1. Percentage of the investment executed before the impact evaluation started (placebo)
  2. Percentage of the investment executed by the end of the impact evaluation period
  3. The difference of the percentage invested in the end and in the beginning of the impact evaluation period
  4. Indicator for a construction finished during the impact evaluation period
  5. Indicator for a construction canceled during the impact evaluation period
  6. Indicator for an updated conclusion date for the construction during the impact evaluation period
  • CONSORT

Below follows the CONSORT chartflow:

  • Intervention 1:
  • Intervention 2:

Intervention 3:

  • Reasons for exclusion in the CONSORT:

    • Intervention 1: Constructions that were not in the App when the intervention started.

    • Intervention 2: Constructions that were not in the App when the intervention started.

    • Intervention 3: Constructions that e-mail contact information was not provided by the Ministry of Education website.

  • Statistical analysis:

For all interventions, we run the following regression model:

\[ Y_i \quad = \quad \alpha + \beta T_i + \gamma X_i + \theta F_i + \varepsilon_i \]

Where \(i\) indexes a given school observed in the intervention. \(Y_i\) represents an outcome variable. \(T_i\) represents a treatment indicator. \(\beta\) represents the estimated Average Treatment Effect. \(\gamma\) is a vector of pre-treatment coefficient effects and \(X_i\) a vector of pre-treatment covariates. \(\theta\) represents a vector of fixed effects estimands and \(F_i\) the Brazilian state level fixed effects indicator vector. \(\varepsilon_i\) is the common error term.

We run three types of analysis:

  1. Full regressions with all data available
  2. Full regressions using Inverse Probability Weights
  3. Regression where we match the control group with a same-size treatment group on the proximity of covariates using Genetic matching.

All models use municipal-level cluster robust standard errors and state-level fixed effects. For all models, we also run regressions without clustering and without fixed effects.

  • Software: We use R version 3.6.0 (2019-04-26). For the regression models estimation, we use the package lfe. For the Genetic matching, we use the package MatchIt. Everything in this report is fully automated and can be reproduced using R Markdown.

Further information about the workstation follows below:

sessionInfo()
## R version 3.6.0 (2019-04-26)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: OS X El Capitan 10.11.6
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] parallel  stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] officer_0.3.5        flextable_0.5.5      rsvg_1.3            
##  [4] DiagrammeRsvg_0.1    DiagrammeR_1.0.1     knitr_1.23          
##  [7] DeclareDesign_0.18.0 fabricatr_0.8.0      ri2_0.1.2           
## [10] estimatr_0.18.0      randomizr_0.18.0     MatchIt_3.0.2       
## [13] stargazer_5.2.2      cobalt_3.7.0         lfe_2.8-3           
## [16] Matrix_1.2-17        abjutils_0.2.3       brazilmaps_0.1.0    
## [19] lubridate_1.7.4      devtools_2.1.0       usethis_1.5.1       
## [22] compareGroups_4.1.0  SNPassoc_1.9-2       mvtnorm_1.0-11      
## [25] survival_2.44-1.1    haplo.stats_1.7.9    haven_2.1.1         
## [28] dummies_1.5.6        forcats_0.4.0        stringr_1.4.0       
## [31] dplyr_0.8.3          purrr_0.3.2          readr_1.3.1         
## [34] tidyr_0.8.3          tibble_2.1.3         ggplot2_3.2.0       
## [37] tidyverse_1.2.1     
## 
## loaded via a namespace (and not attached):
##   [1] uuid_0.1-2          readxl_1.3.1        backports_1.1.4    
##   [4] Hmisc_4.2-0         igraph_1.2.4.1      lazyeval_0.2.2     
##   [7] sp_1.3-1            splines_3.6.0       TH.data_1.0-10     
##  [10] digest_0.6.20       htmltools_0.3.6     viridis_0.5.1      
##  [13] magrittr_1.5        Rsolnp_1.16         checkmate_1.9.4    
##  [16] memoise_1.1.0       cluster_2.0.8       remotes_2.1.0      
##  [19] modelr_0.1.4        sandwich_2.5-1      prettyunits_1.0.2  
##  [22] colorspace_1.4-1    rvest_0.3.4         rgdal_1.4-4        
##  [25] pan_1.6             xfun_0.8            callr_3.3.0        
##  [28] crayon_1.3.4        jsonlite_1.6        lme4_1.1-21        
##  [31] brew_1.0-6          zoo_1.8-6           glue_1.3.1         
##  [34] kableExtra_1.1.0    gtable_0.3.0        webshot_0.5.1      
##  [37] MatrixModels_0.4-1  V8_2.3              pkgbuild_1.0.3     
##  [40] Rook_1.1-1          rms_5.1-3.1         jomo_2.6-9         
##  [43] SparseM_1.77        scales_1.0.0        DBI_1.0.0          
##  [46] Rcpp_1.0.1          viridisLite_0.3.0   xtable_1.8-4       
##  [49] htmlTable_1.13.1    units_0.6-3         foreign_0.8-71     
##  [52] Formula_1.2-3       truncnorm_1.0-8     htmlwidgets_1.3    
##  [55] httr_1.4.0          RColorBrewer_1.1-2  acepack_1.4.1      
##  [58] mice_3.6.0          pkgconfig_2.0.2     XML_3.98-1.20      
##  [61] nnet_7.3-12         labeling_0.3        tidyselect_0.2.5   
##  [64] rlang_0.4.0         visNetwork_2.0.7    munsell_0.5.0      
##  [67] cellranger_1.1.0    tools_3.6.0         downloader_0.4     
##  [70] cli_1.1.0           generics_0.0.2      broom_0.5.2        
##  [73] evaluate_0.14       yaml_2.2.0          processx_3.4.0     
##  [76] fs_1.3.1            zip_2.0.3           mitml_0.3-7        
##  [79] nlme_3.1-139        quantreg_5.41       xml2_1.2.0         
##  [82] compiler_3.6.0      rstudioapi_0.10     curl_3.3           
##  [85] rgexf_0.15.3        e1071_1.7-2         testthat_2.1.1     
##  [88] stringi_1.4.3       HardyWeinberg_1.6.3 highr_0.8          
##  [91] ps_1.3.0            desc_1.2.0          gdtools_0.1.9      
##  [94] lattice_0.20-38     classInt_0.3-3      nloptr_1.2.1       
##  [97] pillar_1.4.2        data.table_1.12.2   R6_2.4.0           
## [100] latticeExtra_0.6-28 KernSmooth_2.23-15  gridExtra_2.3      
## [103] writexl_1.1         sessioninfo_1.1.1   codetools_0.2-16   
## [106] polspline_1.1.15    boot_1.3-22         MASS_7.3-51.4      
## [109] assertthat_0.2.1    pkgload_1.0.2       chron_2.3-53       
## [112] rprojroot_1.3-2     withr_2.1.2         multcomp_1.4-10    
## [115] hms_0.4.2           influenceR_0.1.0    grid_3.6.0         
## [118] rpart_4.1-15        class_7.3-15        minqa_1.2.4        
## [121] rmarkdown_1.13      sf_0.7-7            base64enc_0.1-3    
## [124] epitools_0.5-10

F. Other information

  • IRB: FGV exempted the primary investigator from getting IRB approval because this research mostly worked online data and was approved by the Brazilian Ministry of Education.

  • Pre-registration: this research has been pre-registered on the EGAP pre-registry tool: (https://egap.org/registration/4505)

  • Replication materials: The replication materials for this project are available here: (https://github.com/umbertomig/TDPImpactEval)

  • Funding: This research received funding from the Google Social Impact initiative. Google did not interfere in any aspects of the research design and analysis.

  • Conflict of interests: The authors of this analysis declare no conflict of interest.

Intervention 1

Percentage concluded before the intervention started (placebo test)

## 
## Percentage Concluded Before (placebo)
## ======================================================
##              (1)         (2)        (3)        (4)    
## ------------------------------------------------------
## ATE        -1.266      -0.536      -0.187     -0.289  
##            (2.353)     (2.331)    (2.413)    (2.398)  
##                                                       
## FEs          No          No         Yes        Yes    
## Controls     No          Yes         No        Yes    
## N           2,352       2,352      2,352      2,352   
## R2         0.0002       0.014      0.029      0.040   
## ======================================================
## Notes:          ***Significant at the 1 percent level.
##                  **Significant at the 5 percent level.
##                  *Significant at the 10 percent level.
##             Cluster-robust SEs at the Municipal Level.

The placebo test remains not significant, as expected. Note that we are using a p-value of 10%, adding extra sensitivy to the hypothesis tests carried out here.

Percentage concluded by the end of the intervention

Next, consider the percentage of construction that was carried out in this phase of the impact evaluation. The table below displays the results, presenting a null effect of the app over this indicator.

## 
## Percentage Concluded
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -2.721         -2.102        -0.345        -0.974    
##             (2.581)        (2.484)        (2.537)       (2.482)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,352          2,352          2,352         2,352    
## R2           0.001          0.023          0.044         0.053    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Delta Percentage of Execution

We take the difference between the inicial and final percentage of the executed construction in this impact evaluation. This aims to capture changes in the construction pace, that the app could have positively impacted.

## 
## Difference Percentage Begin and End
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -1.455         -1.566        -0.158        -0.685    
##             (1.307)        (1.228)        (1.276)       (1.261)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,352          2,352          2,352         2,352    
## R2           0.001          0.046          0.062         0.084    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Finished indicator

A finished school in the period was labelled the schools that were still not concluded by the begining of the intervention, in August 2017, but that were finished by March 2018, when the intervention stopped. The regressions here are linear probability models, and the coefficients can be interpreted as improving the chance of XXXX.

## 
## Finished Construction Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE         0.066***       0.064***      0.073***      0.069***   
##             (0.008)        (0.007)        (0.010)       (0.010)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,352          2,352          2,352         2,352    
## R2           0.006          0.012          0.024         0.029    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

In this indicator, we witness a strong and consistent effect around of 5% impact. This represents that the presence of the App increased in around 5% the chance that the construction finished.

Cancelled indicator

A cancelled school in the period is the schools that were still not concluded by the begining of the intervention, in August 2017, but that the construction was abandoned by March 2018, when the intervention stopped. The regressions here are linear probability models, and the coefficients can be interpreted as improving the chance of a cancellation, whenever the coefficient is positive.

## 
## Cancelled Construction Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          0.001          0.001          0.001         0.001    
##             (0.001)        (0.001)        (0.001)       (0.001)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,352          2,352          2,352         2,352    
## R2           0.0001         0.002          0.006         0.009    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Again, the results are significant, and around 2.8%. We interpret this result as the App pressure making the policy-makers more resolute. They either work toward finish the constructions, or they report them cancelled, voiding the impact of the App.

Updated end date for the construction

This indicator reports whether the construction date was updated in the Federal Government databases. An update can mean a change toward a more realistic conclusion date, ameliorating the expectations of the citizens. The results below show that there were no changes in this indicator.

## 
## End Date Update Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          0.036          0.037          0.061         0.059    
##             (0.041)        (0.041)        (0.042)       (0.041)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,352          2,352          2,352         2,352    
## R2           0.001          0.016          0.061         0.076    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Intervention 2

Percentage concluded before the intervention (placebo)

## 
## Percentage Concluded Before (placebo)
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -0.693         -0.593        -0.625        -0.603    
##             (1.635)        (1.611)        (1.649)       (1.631)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,304          2,304          2,304         2,304    
## R2           0.0001         0.029          0.043         0.055    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Placebo not significant, as expected.

Percentage construction concluded

## 
## Percentage Concluded
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -0.146         -0.025         0.029         0.046    
##             (1.696)        (1.675)        (1.712)       (1.699)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,304          2,304          2,304         2,304    
## R2          0.00000         0.037          0.060         0.076    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

App did not affected the percentage concluded. Moreover, the sign is flipped, suggesting that it could potentially have a negative impact on the percentage concluded.

Difference between begin and end of percentage concluded

## 
## Difference Percentage Begin and End
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          0.547          0.568          0.654         0.649    
##             (0.609)        (0.612)        (0.615)       (0.619)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,304          2,304          2,304         2,304    
## R2           0.0003         0.015          0.036         0.045    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

No effect in the difference between begin and end.

Finished construction indicator

## 
## Finished Construction Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          0.002          0.001          0.004         0.003    
##             (0.015)        (0.015)        (0.015)       (0.015)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,304          2,304          2,304         2,304    
## R2          0.00001         0.011          0.024         0.031    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

No impact in the second phase of the app on the finishing of a construction.

Indicator for cancelled construction

## 
## Cancelled Construction Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          0.002          0.002          0.002         0.002    
##             (0.003)        (0.003)        (0.003)       (0.003)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,304          2,304          2,304         2,304    
## R2           0.0001         0.013          0.011         0.023    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

No effect of the App on cancelling of constructions.

Updating end date

## 
## End Date Update Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          0.000          0.000          0.000         0.000    
##             (0.000)        (0.000)        (0.000)       (0.000)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            2,304          2,304          2,304         2,304    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Small effect on update, in the model without Fixed effects and with controls. Very weak evidence to conclude anything concrete here.

Campaign Impact Evaluation

All covariate balanced. Note that the data is not sufficient to compute the differences in the campaign for the FUNDEB transfers indicator.

Percentage concluded before (placebo)
## 
## Percentage Concluded Before (placebo)
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -1.722         -1.895        -1.842        -1.711    
##             (2.017)        (1.997)        (1.972)       (1.969)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            1,855          1,855          1,855         1,855    
## R2           0.0004         0.042          0.086         0.092    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

No effect of the campaign on the placebo.

Campaign impact on percentage concluded
## 
## Percentage Concluded
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -2.082         -2.227        -2.253        -2.075    
##             (2.079)        (2.057)        (2.018)       (2.018)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            1,855          1,855          1,855         1,855    
## R2           0.001          0.053          0.100         0.111    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

No effect of campaign on percentage concluded.

Impact of campaign on the difference between begin and end percentage concluded
## 
## Difference Percentage Begin and End
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -0.360         -0.332        -0.411        -0.364    
##             (0.694)        (0.690)        (0.692)       (0.691)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            1,855          1,855          1,855         1,855    
## R2           0.0002         0.021          0.025         0.041    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Small negative effect in the uncontrolled models. Not strong evidence enough to conclude that the campaign affected the changes in percentages reported.

Impact of campaign in constructions reported finished in the period
## 
## Finished Construction Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          -0.016         -0.016        -0.016        -0.015    
##             (0.017)        (0.017)        (0.017)       (0.017)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            1,855          1,855          1,855         1,855    
## R2           0.001          0.031          0.033         0.045    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Again, very small evidence on the uncontrolled models. Suspicion: the controls have missings in particular points that make the models insignificant. Suggestion: run Amelia, or another missing data simulation device to check for the possibility of differential missing.

Campaign impact on constructions reported cancelled
## 
## Cancelled Construction Indicator
## ==================================================================
##               (1)            (2)            (3)           (4)     
## ------------------------------------------------------------------
## ATE          0.001          0.001          0.001         0.001    
##             (0.001)        (0.001)        (0.001)       (0.001)   
##                                                                   
## FEs            No             No            Yes           Yes     
## Controls       No            Yes            No            Yes     
## N            1,855          1,855          1,855         1,855    
## R2           0.0002         0.002          0.003         0.005    
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Here the results are strong and robust. The campaign reduces the amount of constructions reported as cancelled by around 9%. This represents an important outcome in terms of constructions, as fewer constructions were abandoned as an effect of the campaign.

Campaign effect on enddate updating
## 
## End Date Update Indicator
## ==================================================================
##                (1)             (2)           (3)          (4)     
## ------------------------------------------------------------------
## ATE         -0.000***       -0.000***       0.000        0.000    
##              (0.000)         (0.000)       (0.000)      (0.000)   
##                                                                   
## FEs            No              No            Yes          Yes     
## Controls       No              Yes            No          Yes     
## N             1,855           1,855         1,855        1,855    
## R2          -Inf.000        -Inf.000                              
## ==================================================================
## Notes:                      ***Significant at the 1 percent level.
##                              **Significant at the 5 percent level.
##                              *Significant at the 10 percent level.
##             Cluster-robust Standard Errors at the Municipal Level.

Small effect, specially in the controlled and fixed effect model. However, the effect is small and we advice caution in interpreting this as an evidence that the implementer changed the dates for a better estimate.

Descriptive stats

# Visible App Begin
tab <- data.frame(table(impEvalph1$visAppIni))
names(tab) <- c('VisibleApp', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=VisibleApp, y=Frequency)) +
  geom_bar(stat="identity")
p

## Joining, by = "City"

## Joining, by = "City"

Alerts

Analyzing the campaign alerts

Alerts by Date

tab <- data.frame(table(allalert$answeredBy))
names(tab) <- c('AnsweredBy', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=AnsweredBy, y=Frequency)) +
  geom_bar(stat="identity")
p

Alerts by construction

# Number by construction
tab <- data.frame(table(table(allalert$project_id)))
names(tab) <- c('AlertsSameConstruction', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=AlertsSameConstruction, y=Frequency)) +
  geom_bar(stat="identity")
p

Alert Answers

# Alerts Answered
tab <- data.frame(table(allalert$status))
names(tab) <- c('Status', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=Status, y=Frequency)) +
  geom_bar(stat="identity")
p

Alerts by validity

tab <- data.frame(table(allalert$status2))
names(tab) <- c('Status', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=Status, y=Frequency)) +
  geom_bar(stat="identity")
p

Frequency of Alerts by State

# Alerts by state
tab <- data.frame(sort(table(allalert$state)))
names(tab) <- c('State', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=State, y=Frequency)) +
  geom_bar(stat="identity") + coord_flip()
p

Frequency of Alerts by State by Total of State’s Construction

Barchart with frequency of alerts by each construction. This Figure shows the amount of alerts divided by the amount of schools in a given State. Increasing this number denotes that more schools are

# Alerts by state over total
tab <- data.frame(sort(table(allalert$state)))
tab2 <- data.frame(sort(table(unique(impEvalph2[,c('id_project','state')])$state)))
names(tab) <- c('State', 'Frequency')
names(tab2) <- c('State', 'Total')
tab <- left_join(tab, tab2)
## Joining, by = "State"
## Warning: Column `State` joining factors with different levels, coercing to
## character vector
tab$Proportion = 100*(tab$Frequency/tab$Total)
tab <- tab[order(tab$Proportion),]
tab$State <- factor(tab$State, levels = tab$State[order(tab$Proportion)])
tab
p<-ggplot(data=tab, aes(x=State, y=Proportion)) +
  geom_bar(stat="identity") + coord_flip()
p

Frequency of Alerts by State and Status

# Status answer by state
tab <- data.frame(table(allalert$status2, allalert$state))
names(tab) <- c('Status', 'State', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=State, y=Frequency, fill=Status)) +
  geom_bar(stat="identity") + coord_flip()
p

# Map not answered by state
stateCode <- pop2015[,c('ibge_code7', 'state')]
stateCode$State <- substr(stateCode$ibge_code7, 1,2)
stateCode$ibge_code7 <- NULL
stateCode <- unique(stateCode)
tab <- data.frame(table(allalert$status2, allalert$state))
tab <- tab[tab$Var1=='Not Answered',]; row.names(tab) <- NULL
names(tab) <- c('Status', 'state', 'NotAnswered')
tab <- left_join(tab, stateCode)
## Joining, by = "state"
## Warning: Column `state` joining factor and character vector, coercing into
## character vector
tab$State <- as.numeric(tab$State)
rm(stateCode)
tab$state <- NULL
tab
mapbr <- get_brmap('State')
plot_brmap(mapbr,
           data_to_join = tab,
           join_by = c("State" = "State"),
           var = "NotAnswered") +
  labs(title = 'Frequency Not Answered by State')

# Map valid answer by state
stateCode <- pop2015[,c('ibge_code7', 'state')]
stateCode$State <- substr(stateCode$ibge_code7, 1,2)
stateCode$ibge_code7 <- NULL
stateCode <- unique(stateCode)
tab <- data.frame(table(allalert$status2, allalert$state))
tab <- tab[tab$Var1=='Valid Answer',]; row.names(tab) <- NULL
names(tab) <- c('Status', 'state', 'ValidAnswer')
tab <- left_join(tab, stateCode)
## Joining, by = "state"
## Warning: Column `state` joining factor and character vector, coercing into
## character vector
tab$State <- as.numeric(tab$State)
rm(stateCode)
tab$state <- NULL
tab
mapbr <- get_brmap('State')
plot_brmap(mapbr,
           data_to_join = tab,
           join_by = c("State" = "State"),
           var = "ValidAnswer") +
  labs(title = 'Frequency Valid Answer by State')